IN THE Ul 




D STATES PATENT AND TRADEMARK OFFICE 



PATENT APPLICATION 



In re application of 



Docket No: Q63913 



Rui ISHIYAMA 



Appln.No.: 09/823,763 



Group Art Unit: 2621 



Confirmation No.: 9124 



Examiner: Tom Y. LU 



Filed: April 3, 2001 

For: DEVICE, METHOD AND RECORD MEDIUM FOR IMAGE COMPARISON 



Submitted herewith is a copy of an executed Affidavit Under 37 C.F.R. §1.131 signed by 
Rui Ishiyama executed on July 7, 2006. Submitted with the Affidavit are the following 
documents: 

Document A: A specification that the inventor himself drafted 

Document B: A Notification of employee's invention 

Document C: A Request for patent application 

Document D: A copy of the case history for the priority document 

Document E: A verified English translation of the priority document 



SUBMISSION OF AFFIDAVIT UNDER 37 C.F.R. §1.131 



Mail Stop Amendment 
Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 223 1 3-1450 



Sir: 



SUGHRUE MION, PLLC 
Telephone: (202) 293-7060 
Facsimile: (202) 293-7860 




WASHINGTON OFFICE 



23373 



CUSTOMER NUMBER 



Date: July 17,2006 



PATENT APPLICATION 
>N THE UNITED CTatk PATENT AND TRADEMARK OFFICE 

tn rc application of 

Docket No: Q63913 

Rui fSHlYAMA 
Appln. No.: 09/823,763 

Oroup Arc Unit: 2621 

Confirmation No.: 9124 

Examiner: Tom Y, LU 

Filed: April 3, 2004 

** dev1c ^tho D an DREco ^ 

AFFIDAVIT UNDER 37 CVJL § 1 13 1 

Lornreiisioner for Patents 
P.O. Box 1450 
Alexandria VA 22313-1450 

Sir: 

», Rai bhiyama. hereby dccl are ^ $la , c „ fo „ ows . 

' • ' - *c inventor named in the abovc-c aptioncd Us . AppljC3lionNo 09/823 m 
Apn. 3. 2001. which Apriority to ^^^^^.,05399/2000. 
2. At the time I invented the present invention, I \jv as employed, by NEC 
CORPORATION (hereinafter "NEC"). 

* Prior to March 30, 2000. the U.S. FUi„ B Date of U.S. Patent No. 6.956.569 to Roy 
« 1, «hc invention „ durfb* ,nd C | aimcd in the >b,vc TcferCnC ed ^fafa, « cornpleicd 
at NEC. as evidenced by the following: 

1. Prior to March 30. 2000, having earlfc, conceived the idea aS s« forth in the 
specif of the above ^tcnctf nppJi^.thcp^entinv^tion f 0fma Uv submitted 
in the form of a specification as show, in Exhibit A, as written by fte invcnlor , (0 [hc palenl 
d, Pa n menl Nfcc in form of .NoU^icn of cmptoy^ invention, as shown in Exhibit B. 



AIJ1DA V1T UNDER 37 CFR $ | .13 1 
U.S. APPLICATION NO. 09/823,763 

NumUr 33S-09734, a M p, of wH jch is ansckell tetl0 

1. e« blt Ajsthe 22 . plee iBcd(iMi0 „, ^ by „„ ^ |k t)()ht w;ih ^ 
«N«» — -f-M s^oo. Bxliibit y\ includes MWUI1 , su „ on , „ 
•Utal «. 13-15. 2.-22. K27.2,, „. ,5, 37^0, 4M S . S M 3 , 55, S.-S0, «, «„. „.„ 

" " ** f< " towin8 ° f «* docun,™ i„ Exhibic A <tah„ 

>«ppon may also be found clMwbtrc I, Exbibil A): 

Claim 2 - Pages 34^39, FJGS. M 

Claim 4- Pages 34-35 

C\ Aims 6-9 - Pages J 6-3 8 

CUjms 13-15 - Page 36 

Claim 2 1 - P a <.cs 39-4 1 , FIGS. 6 1 

Ctaim 22 • Pages 20-21 . 39, 4 J (citing pages 30-34 and 37-38) 

Claim 2^ - Page 9 

Claim 27 -Page 35 

Claims 2R-29 - Page 39 

Claim 30 -Page 4 

Claim 33 - Pages 34-3? v FJGS. 1-5 

Claim 35 -Pages 34 JS 

CUirn 37-40 -Pages 36-38 
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U.S. APPLICATION KO. 09/823,763 



Claim 44-46 - Page 3(5 

Claim 52 - Pages 39-41, FIGS. 6 and 7 

Claim 53.- p a g« 20 .2l , 39, 41 (eking page, 30-34 37-38) 
Claim 55 . Page 9 

CUim58-Paec35 
Claim 59 T 60 - Pag$ 39 
Claim 61 - Page 4 

Claim $4 - p agcs 34-39, FIGS. 1-5 and page to 

Claim 66 -Pages 34-35 

Claim 68-71 -Pages 36-3$ 

Claim 75-77- Page 36 

Claim 83 - Pag«s 39-41, FIGS. 6 and 7 

Claim 84 - Pages 20-21. 39. 41 (citin f p ages 30-34 and 37-38) 

CUim 86 . Page 9 

Claim 89 -Pace 35 
Claim 90-9 U Page 59 
Claim 92 - page 4 
Claim 93 - Page 4 
Claim 94 - Page 4 

CUim 95^ Page 3 S * 
Chiiru 96 - Page 39 
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AFR DAVIT UND£R 3? 

US. APPLICATION NO. 09/823 jll 

Claim 97 - p a g e 4 
Claim 98 - Page 4 
Claim 99 - Page 4, 

employed by NEC. 

.* m^ ilny BJM , WIi£ati „„ rtk , he ^ ^ ^ ^ ^ 
o*. of nec 10 „, ™.« m,™,™, A „ Pr , f „,„ , tw ^ h ^ _ Exhjbi( c 

and appravu) on March' 28, 2000. 

I» the ordinary cour« of busing and in due course, NEC r^ed and ^d 

200W05399>va $ f,| e < J . onA p rij3>2000 

10 In view of thc fQregoingi ■ . js dcar ^ ^ flanied ^ ^ 

capuoned applied, invented fl* subjCC . matlcr 0 f thc eWpB prior to ^ March ^ 200Q y s 
filing dale of U.S. Pucm No. 6.956.S69. 



AFFIDAVIT UNDER 37 CPR §l.)3 J 
U.S. APPLICATION NO- 09/823,763 

1 hereby declare further (hat all statements made herein are of my own knowledge and „ 
iruc and that all statements made on information and belief arc believed to be true; and further 
that these siaicincnls were made with the knowledge that willful false statements and ihe like $o 
made arc punishable by fme or imprisonment, or both, under Section 1001 of Title 18 of Ihe 
United Slates Code, and thai such willful false statements may jeopardize the validity of ihe 
application or any patent issuing thereon. 
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i. w^iff-sfctocD^fro^-^fe^-^^-^K: a«r *a«aet*$i*-c, 

bf?*** j< ?<j5©»fc§SI**fliV^TA;&Bfci: LT22»C7Gl8«fc4SIUBU 

®^£ai£SnT^33&7C^:|*£:£fj2£7 r --:$' fcffl^T^fiScU 
^M^^i^Jili^^^A^lli«tS i b^v^^ii^^*^6TJt^!li«^ LT£jSU 
ttJfcttSfcklfflEA^IBfllioaittKofffliiifltiW-JIU, 8*JH£lA0l¥flHBc£ o*v>t, 

37ili«»r**>iav^H«*it«IB*i:LT*jBR-rstj 

3. nragrc iBiiosauis^ftKife i>x, mmmwmwj*£ffl(D£.r&t>K 



0, 

"1 L ffi # T & 4j © T£> 5 



<D± tit*, 

/*K*JV*T, 8*a»SnTV>6fl5liHSC»fflnaS:5pIffl t5 
6. j»#*3 KWWHflftJWa-^ft Cisl^T, BtTilEJSAae»r*3tt4HHiWffi»jffi|ftffPeo±/a*5«J:u f */a' , <^ h A- 



7. »*81 *^ !if*86 IB 4? OH*«R^acKfeV>T. 

8. ff*8l ff*S6 E*o fifcHH^ifeCfcVT, 

9. ff*81 »*88 Eft ©H«M«£-2fifeC:|3^T, lfflES«»SC:^TRIt**iBl5rr*ftfc>»)ir 

l^ts\^L&&<D31k7cMW;&Wlfe-f5>m?-Z. ^1^©3#C7C?8M*&^ L4>&© ZftLTm^oy^t ft <53#; 

i*%fr? k©T£>9. UR©*lSi£*5l^T»;i:3»e7C »ttfcLTtfrE©W©HMft^-*/£tt*fl3V*S 

12. St&Sl 3**811 *t? CEHfiDHflUB^SsKfe^T, 

3&7C7EM*© ffi!l££fTfr-f , 3J^EIffi^*^3^7c:^^ A2> L T ft < 

13. If Ml A>5> 11*812 £ -CCl2*©HfcJia£*ifeK:fe^-C, tfrl5Stt^.©ffl^5:?f tof, JMttHffiefr 

14. !if*81 A-?, ff*813 ifCE «©HfcJJRS-2rifeKi3^T, 

3 £ mm £ T Z> SMAv&ft. 

15. |f*^l j6^?> lt*814 *-CCE*©H(fe f8£-^j£K:m>T, S»**ilWEfl8-6-S*fti:r«ri:*4*»i: 

16. !i*81 ^e, 11*814 £-Ct;:E«©B& JS#^ftC:fc^T, AMoiBfeKrERa^jftfc-rsi 

17. M&*ftotz&(D&tyte(D7-*-? *y-'--9 -<-X Cfgf Ste^fttf, 

fIie^fr©3^7C^©#fii:Bt*3lt'5«@l©S:i : f^5:iBiJSL, ii(JK^{*:©£ ft^t L T Hi^-T -SStt^ 



* y ^ & if ©A^B&^iai* l t mtrr * mgL^&t , 

JkU !WEffiB*»fcRfitBI^S5»£*>oTfrE M^A#^®MW^#0!)TTCDiaft«;ASKXIbBftP 

t LT±js8-r*iSfc*ja fat, 

ttMW*»iiiflfc**»6l^rcA*afl^^r*tifiv^ll«sr*»TJt^^s«fc txtu 27-f a ^ at 

ifcixf ail> 

18. »#B17 E«fflH«JH^*Btr*v>T, ttEKte^K**, 

«Tl5^^IEi!j^F^rttr2boTffieA*iii^(r*^5a^iS^%ii:^iii^^LT^ J5&-rs]Kl9I&#jft£^K 

B&Kl||*;,fil\iIifgT£>-5itf£iIi^ : &:£j5£-r5 

20. m&mn E*fflHBjtR^SBK*s^T, mmmiE^ a* 1 , 

«rE«W«ftiBflki¥*±ja»»W4^ottif-«i#WS:l|-ri4:K:J:0, ABBA #c«fc 9Bft±C£ftTV> 



21. H*«17 CEttOBiBJgl&SBfr *5^T, 

«rEJH^s»r*iit*jiRw*#«t3eM*«» irEB£t£n-c^&jgi<H£ft'igfe # 

w B. 

22. s»^clgl8 (zE«S©H«M£- KBfcfc^T, 



23. is*Jii9 izumcomm ^-sb^^t, 

24. W*J|17 i»& si*JS6 *T?KE*©Hftlia£-StRK*5VN7\ 

if f e A27 5 nfc * ffl v ^ -s t © -c afe -5 

25. »*«17 frh m-&m *T? irE*©B«MR'ft*B£*iV*"C, 

WEWB^H^&lfjaSftSAAIiBfeA^i: U ttHBt^oTV^Ifofl:© tfcfiSM&*ift5£ Ltt&BSF 

PS % its * f -5 feB^§§Jt^3=@: 

26. MMtgm ^p, it #9(24 s x*iznzm<z>m&M&mmzi3^x^ iifsc^tti^i^s^, 

9^«cfiRn«:l9:BLTIBft11lff«:IRKLttlSM4B«:Sjgt4S2: LTd^T £B g>^S?#J£ 



27. »#£17 !f#iH24 5 -e£E*©HBJH£*BK iiUT, S£#ts£«£3Ma# % 
Bfifei: LttHJt5i«*^S 

Hfc£j«*SA*, JHWAfl : S:S!EftS-ti-4ftto*)K:iKrE7 l ^^?- + iafllSS:¥iJffl LXBf^Sltt* 

28. If #£17 af#«24 *T£E*©B»J*W&-SsBK*5^T, iEM*Sift«, 
loft^L'>&©3&SJEM*£A#i: U ttlo©3afe7cEM*fcV^sl>ft©3ifc7C JP*t©¥*S»i:<ca3*7C»tt 

fTET-^Eti^SA 1 , HrE^fiRSnfcW»«©**Et«-5t,©i?2bO, 
Ha*#«:a«1KrE 3 F^»tt*5PJfflr « *>©-?&£ 

29. 8»#«i7 m^mzi i:-e»zE«©@«JlR-d-StB*=:fev^, 

iTE3»:7C»ttjB!J3e*Si**: < , SU34H®**»'&3»C7cJeJtt«:A>!7'r 5 
C t tti BBJ816-&B. 

30. J»#JB17 8f*JB25 * «t l>* If 5)^11 1 A> ?> If #£12 $ -CKE«©ii»IH-e -£BK: fcV^T, 
ffiES*f*iUJS^S*«3tc < . 0J&BlilP¥*» f>S«$4ASt s 



nu leaa (3 *5i n t l tz © fee it m * mm r * 

^ §/j jp^j 

33. 91*3117 If*^30 StCKIEB ©H^M-S-sggC^T, AH©g£j|frf2j8^&£T-& :i i: 

34. »*si7 a*Ji32 E*©B*jiS£-«iiK:*5^T, xnm^n^t ^w^^.£ lx 7 

W«»**«*JS*S«iA*«*fflV^scfc*W«^:■rsBI«JW■fr I Bo 

E««E#cHr lt^*-^,, * LSRr;ri^-fcrA,;6 ? ?§«LT v^fc/fw-fttf^t-r. 

H»M#S«i:a: [05] c^UfcJ:-5K:, 3ifc7c3SM±fca3K: EB*n;MM*;&M»nJ**©**, rt^frir 
£-*fjfc*E«LT33<a«ifi6*:, A£$nfcB&(#)»rra;&s»i£;snT^.2> ^^lir-^fcittl'&Lt^ 

<ra^fflg©~-3i&»e>fl*ricSft-cv^„ -^n ^©jflSfcfcv^sMfcSftfcBfctt, 2 ^yttfjt£jK^K>-^m-r>2 

£CJ:9, 32*7cJPttfc*fKSHIlL-C?r.M!-r*. Sfcfc©® feftd-&ftf*, JATT:fc|R*#J«Lfc#&«9ir4. 

Z^B«=fe»<iL-r*i*A*k tT2»:7cBflR«rfflV^*g«flH■6■a^S©— «fc |_T. UfciSK (2) 0l$fffg2872776^ 
r«Bfc*l*36«J ) TwSftS&WtffcS. #ft«ttB*ttfefcLTAIHJ©B*MLTi59, [013] ©<fc 
■5«c oTV^o 8«9tt * y 5 1 1C i 0 l*Snfc2»:Sit*, ISlit# ®12trgeiS LT* < . 

^ f £z ]3x ' z ' {!k7c0)mm ^ ;K:,:3m ^ tLX jES-fk^aucJioT, lttlEA#fifcj&»&B*>*©flfc 

«Lt£;te^T2 »:7c«i«:ffi«4:*#*0iE*ft*fi-afcjE3ftfl: B&£ffi;fr-f -So ftft£BftJ£«^gl5 trio 
TC*#«12*»6»*fflSix *a«B«fc«re:E»*fcBfc*/<*-:/B«««fK£9ifc«U ffi-frJM&&tti * 

Sfc v 3^7t^JrfflV^ctie5f£©^S^©^Ji: LI, XIR (3) (»M*F 9-259271 rA*J»^«Hj JtSSnsa 
f^iib^^R"^ USM] ©«fc-5fc*/sS*i:*. 3MI«i:ii, 3&7C«fct** 7-«*BtfjBI #S21c«tr)« 
£*fft* &©3&^MSfc*5-11t$ft£8iJ£U E*^S22C K«LT*5<„ JStt-BSCfc, 3*tc»«* 
ItfflJM^fr «fc OAir- *i:LT3*7B»«i:*9-»«*B3eu WWfc ■ EHE^«24K:J: 9, ml IEA* 

«25KJ:oTl»26©*/J\ffl*:fc«>s;; fcTJ32te7c»fcMR»©MiE*frV^ tt*/JxjRSiK:afi-^VvT:jiR^*ff -5 

o 

AiS(5)(#M s F6-168317 rflaAs&SiJiggJ [015] © £ -5 SrfcO, a£iB# £: j&-£-B# i: t, K2 #cteB 

S%Ift5)i^gitife5„ Ctt*.*541£«fc9 21*7cBft*»|BU 4#«»m#«42»:*5VNTMSf 

J»© *&*B*tfc«**fflLT«»j«ffi«*HJ*U aEW«43i:E«-r*. BtCfi, 544tl J: OA 
#H&£: LT22fc7cBft*JHBU.4$«illitii#«45 ^r^iV^TJEPS[^liJ©Ag•>i®^fea : S^^{±l LTftijftttl* 

a*L, M-^^a 47cfeVNTiwiaa»snTv>4#«jAte«tiMriEA*B«ffl#*^fii«*jti(it tsitfi 

^■5r^ToTV^i, 0 d©tSB«J^aft#©ffiB^»»©3C» ! S:» 4X"T 5 fc«>»r, ffi«»»jE3R'fl:#«!46ca3l,'»T 
B*W5fe*#3»:jc»tt ; Ef f A' %*6A»C«)ffla:UTfe*, tta*3»:3CJEJ« J E7 f ^*f!|fflLTffia»»%iE % 

J^TcB^ff tt : &ffl\, , >?)S^I5i: LT, XSS(6) ( TVisual Learning and Recognition of 3-D Objects from Appea 
rancej , Hiroshi Murase and Shree K. Nayer. Int. J. Computer Vision, vol.14 pp. 5-24 1995) A*ib •?>„ 
^©tJ^T'fi [B16] ©J:-5tt*J5Sft«l»3, «^S71liJ:»). »«-r*#«ff t:-^v^TA*B«C*^V^T#^ 
P»n?.*e. »J>**»*«W*#«:IH«Lfc-y->^H«»«:IH^L, 4-«#:tt**«72K: £9DiriEBfe#©k 
^bSr-r-^as-a-S J:-5«rajSB<feW«:il«^»WCJ:0*«), ttXfiBftPh©ffiH«4»«^-r«4$fRSM«r^lK 



^#:©fe«S^A5H^Tfe5^^©M^^#(riS2^7cH^©^'^b^roV^X ttXlK(7)( TWhat Is the Set of 
Images of an Object Under All Possible Illumination Conditions?] . Peter N. Belhumeur and David J K 
riegman. Int. J. Computer Vision, vol.28, pp. 245-260, 1998) lifc^Tf^ L < ##r£ ntWSo #Jf£<7){ftB 

l!frfe©##f£g-3£\ UlE:£gK(7)-e«:I 1 lumination Subspace Method (I^T^ $£(8) T*#fiiR-r&) tflfHS 

m&Tfo ^extreme rayfc«*Hrft*S«R#*£jSU 54Cl5firr4 0 BA-HSKtt*.* 555£ J: oT2&te 

:t©J&W:&#©TT0B&tiftfrieextreie rayfflPO^tSA'IET&SiS^^di: LT^^bt^©?, i&^»S¥£, 
AKfc&fcV^^-3*#0T-C(a*/J*-* &*ffiV^TttJH-Swi:i«-e*5 0 J»iP!MjHM9:56K;fc^T, ire A 
W=fefrV\ :K3:ofcfK*»«:JI!^fcextreBe ray ScDSUgfllKlJ: 9 A#BB£: H 

©B»T**ifctfcBft*£j5S-r4. m&itm^&si \z ts\^x* KJ±tga^hnifsA*B®©«fii«srft»:-r§ 

:£8iK(10) ( H i lumination Cones for Recognition Under Variable Lighting:FacesJ . A. S. Georghiades Pro 
c. IEEE Int. Conf. CVPR, pp. 52-58. 1998) -C&;^£(8) C*5l>T extreme ray&ttg-f ft«i&H|i&if 

tt-a*lJl*fr-3;&ife«: jKUTV^. inK:«l:»)J^tt*«a-e«:V^tto4fcftK:«,*ife(8)4«affl-e*4 LTV^ 

o 

fffIBi:fi^(7) "CliSampl ing Method (UiT^ft (9) t. LT&M)b. LT [B18] © «fc-5fcflM5&Bl4;&ft<l>»3g 
LTU.5 0 fflff£©^fj£(8)©«}: 3 td^T© extreme raySrfr^-f <& £ te^KWA^SOD-C, agl^l;, ^ 

^ik^t. [H4] © @i ficDft&tftez^ < 9MNBc£#&& -5 «t c fr»©B 9i;frra] 

JSUTBBS&aHBU ttBftS * extreme ray LTftffl-fSo i3lP*tt^ij*(8) 

* f * $ £ $ * ^gib^-HtfflJEt- SZLhA'T'^TV - *^ V ft, «ffl«H^*#C PS 6 til L £ 5 i: V n -5 Mfljft 
^(2)tiem$nTV^5J;d^^^52^7cB^|5l-±T-C0M^-a^-Ca, M-o- StfftfcfcStt#03Jfc7cfl&ftro|*s 

y£nTV^j:dfr3&^#tt9J^ Stf&Sfc&O, *Bi:&oT LSE-JMHtf&^fc. rnB, S£i$£A 
»jjr«\ tt«cco«Bf■eA*B«^:»ll^LB**^TV^fcv^»*<t^^K:(«^K:MH■T^ *fc»tt®ffw*r 

XlK(5)^r^^nTv^ft «t-5«c, BKX»©**fcB*fl)*&fcm-rs:£ifett, 3^^^^^^^^^^^^ 

Altt(6)©g«fett % A^BfftOBWAfrfcL-caftoatB-^ISift*®*^** «:BW*#=fe#jfr*4:, 3ri 

*tt(8).Sj*(9)IS«0*ftrtt»fi*<*:C*5B[©^rifij!ft»e>BiB*aT fcB«*aUI^-r*^»A«ib«9, SBI^ft: 
ttWfclHWStKtfjfcKfcfcSw «#©EKoMB*»&+»iE*3teBM*#ffl»3eA*fflBctt-5«c4roiBI« 
*«* 5o Sfc, *-*fc(8). (9)ISffio^&f±i: tc, ftflccDteB-^SSU^JtfeofcB ^<7)ffiB§?^{3 jolt 

*#»©BM*#©TT?0B«S:»*U, fiCfe L**J-r^»****. Lft*«->T, A*B«C 

^■ffi-(8)cDKWT'ti, ?^ttOffl«ISl3JCEi:Textreme ray^rltWf 4f iK$ f^^Ttii ^ < ©ItSi^Sf 



Z>o Xltt(7)C«tntf, tt#^®©ft«l'<* h/U©-5t>MU&»a:&fc©j&« ?7lfi*i»A-, extreme ray©ft»i 

«©Bfc&W3^fcCtntf&5»fc< fcifctf), a«Utc»tt©— tt Wfr*H*C*tLT£T©extreiiie ray£fr»T 

£-Ktt-*©**aflI1- 4:: fc»iT?£fcV". 

#&#ft£ft£>&V^#©TT*©«:/hrftfe©tmiextrerae ray©'» CW« LT##fc*< ©«■*»* 
Stt-afcl^lHHHA**)*. ^8s(9)©gfln? tt, CCPa^StrJnAT. *f©Sa©»[©»6@fc£flJl'>ntf-W>fc 

*»l«©ifl&tt, 

*3»r, a«H$C!fcKfcf f -*j&«««cfllCTtBT?fcO, 

i: 

3*7£»tti:, $Jte©«W©SW£ HSU tt3»C7£JI*:|*i:K*t*fcie« LT *>< . A#£nsffl«{3*f LTa 

®&-e&5J8 W^I^Jil^ : S^S^^n■rv^^)3»:7C^i:HI^^7 f -^' : S:fflV^T^^L, HE JHWS»Bft»*&' 
AABfcK*fc&V*Bfc*#«>TlbRBfc*: LT^jsfcU ttJtttBftkVTIBAAaftOMMffllfilliffitbfi-ff L 

a»snTv*s*#©$fc(6m^4Mta>*#, #©&!!$: fx #^h-r^ 0 

»*5(2 K*c©#iW«:, St&9!l !e38©#&9§£*5V^, Sfj KifcifcBft©±j«#, luISMW3E»;B^T*[i 5»n 

T£/a-r % t, ©-cabs r i: *&mftz 0 

sfcffiiwosjfi^* Fmftscfc, wwaE»fflw**^rst0 ■e*>.«5, frEtfcttB*0£ja#, fries 

i»*^4 eit©ft9Jtt, tt«iii E«o»iBK:fev^T> «r £jftW£ibBfea¥©£/i&*« fl&isa*asK;&^T 
*&*»i:«)?T^T*i*, tt «Rwa!ibBflfe»*i5tttLT*jgr, iwiEiiR^»s^:m*sifcttBft©£j£K::te^ t, 
ttafisnT^iflawaibBftwsripjfflratc-esbs^kftwak-r*. 

e©Jfc«FBft£j«K::fcV , >-C, tta«SnTl>SJHi!IISSIbffiM*jpj;B f£::i:£#S£i:-f So 
fff#«6 £tt©%9itt« lf*JS3 f£«©?IWi;*3 V^X, fff£M-&ig@{rfcnt.5^^^»ilj^©4^*5«J:U ; S 

•So 

afc/BV^OT-fct fe<, mI^A*Sfe^r^oTV^Sfe#:©fii:gS^ ; S:fJlea^S^aTV^?)3»C7c flfcRfcElt** 

spjffl u Tji^r * c: t i: -r s o 



it*sio te«cD^WT'(i, nr&gi a*e> it^iis KiE*©»iiiiK:&i*4a0»e©K*f*©«ije*, 

-TZt-? ;***Bfc»©iM*CB£ifc*.5wfc, Iff&Hl tt*JS8 CE4t©a9JC:mt&ft[9i&l; 

Ttefr*. If*iai0 i2«©5&9JT-&, ffi-fr&fttt#©Bife©ffR9ift# C«t *SE»*S*-r*MWaE»fflM*4ja 
1 4 o Ct»4* > ^Hfti'tJ* T?S*/£l*©»3fc*©JWW*fr*l9:5tU -tcDflRM*#oT-CIB«flMI* 
» ULTiSttf*, fiaMsSrflHeLfcOavtf*-*^?* »^7CJ:4I|i4 J5Rt3;fcW-iiKl9I*#©Rje* 

Hfe&MMg tSo ::©«£*> 5 >7fcv=. t?a. U-*CH0#W-T»»3l*fc!&*&H*& HOT * & & 

* * c j: o £j« u y->zfji>m®LtLTihjj-rz a jut© Ma at&gn nrags v-Knm&mt.m*t. 

^»ttfc^L^ft©3*7cJ§H*©Wi: fe53»C7cJEMftl-3*tiJ27 U M*#ftfc"fc*£T:©4M*©J§Mft»ii1-W 

(DTxoymmnmM^n^^coxh*), &»©&«»= *jv^T&3#:7t;7BMK£: ltibie©^©^^-*^* 

-Tfcfet. llf&Sll fe«©^-C{i, GfKSi^TIMfcjWHfiLTV''* »*©»^t:tt, £T©*fl:©3ifc7cJEM* 
W3*7cJi*tt©*jaK:fijffl-esa^r«fs©— lt, *©j:-5«c^aca«f!iffl -ets. ^..-ea^o©***:©^ 

-Ti-SC, 2o©4fcfl:©3*5c»ttf f -^*M'Cj«r— »S^S«t-5K:W*» L, £H© .#WiirH&*rffi£# 
}U BflB*aa*:MIHT? #$ft^flH*J;&> L ft #6, #^fB±-e s F^^%fi-#-r?)o .[39] ^atr^-rj: a 

w©^ftl44fcfrffl*S[A«3flJSl±C)&:oT«)affl-eS*. ¥*»ttt*ft ©tec fcfc©# 

uT©^-a-ji@©^aac*5^Tti, f-'-^ieiStms^^Hfjie^^^^Sc^m u, #»<*:©3»:5E»*t7 i, -*K: 

* 5 ^c<, »J3ftHBi**»&3»C7nJeJtt*A*LT*5< ^tftfttti. 

r«c*3t. lf*JBl2 ©^^"C(i, -WfcLT, JH^-*fftk<t*»f*: 4 f X*«iHi«cif«WH©ff*'ra*flc'C*)-5 
tSMC, HW*4»&*f*:©3*7cJ»tt ! S:A*T?t S3 k«rWfflLTV>* 0 

lf*Jgl3 IS«©«WTtt, Sff^l A^?> lt*T811 S ■t?CE*©Hfc«l#:fi8iK*5^T, RJ)**S!l3£«i«**«l: 




TKtoib »#JB13 Se*©*iHT?tt, -MULT. m^MAt-tj: 6^ft#XI£gJ n a a & ^ft^®©#as&©^<g 

ffcte-fe Ht&flM !2«©^^Tti, A#ffl»CfcV^M£-*rftftfl:©*Mft £©tfcB$:«ittIU SkttB£©3* 
7cfl[*«:a«7 f -^A»e>5ffl4c:i:A l -e*n tf, «riH#»j«©H»±©ffcBfc3*7cS**^TA#Hfc+©* 

£>9, — #J i: LT3tSR(15)( TAn Efficient and Accurate Camera Calibration Technique for 3D Machine Vi 
sionj .Roger Y. Tsai.Proc. CVPET86. pp. S64--374. 1986) ££*©#&#* 5 i», - 3 TMiPNfflColAT 

T-ecD**©a«i*±jaT?s*cfc, *j«fctf, i*&BB&£&*j&6#ft%tt©#*fcfflH*#©T-c©iBfeff 

fcJ: K>£.J&Ltzm$L (WTCCHlfti:**) i: A*Bfc*afc*fcHi-fcfcSCW*#T?**£W-tfo;fcOB£.5J: 
*fE 311© >*7x-**^TteB8»©^7y*%M«Lfc9A#r4^iS* jflJffl-CS*. 

i»T?ffi«»»0JSSg*ff-5^j*©— «fc LX\t, ***:M»»-C©JW-fr*r ftfttt©CGBJft&£AU 3-CGH 

*££!*.«) ©titB*A#B*i:CGB»!6»5* tHU WfBi|$m^©&«A s o i; l>jifi < fc£ <t ? &CGBi&£3jisi) 
S ©fficB*»*»**B©^9.rf * a#8ctijpJJB-eS a. 

Sfc, 4#*AffiB*A;&Hfc*»&fctiiU ^WSt^M^teBHfKK^T©* **jpjfli UT*fr©«fc«»»*« 

4C> -5 ab -5 o 3tSK(13)( TAn analytic solution for the pose determination of human faces from a mon 
ocular imagej , Shinn-Ying Ho . Hui-Ling Huang. Pattern Recognition Letters, Vol. 19. 1045-1054. 1998) \z 

£;&©□ 5r©#m^%i^■Sil:j^* s 5:v^{rT^T-e^5 3&:^f©&aH^ : &flJffl LTfeB^ *#»4#ife!&*iaifcS 

Sfc, tf*iSl4 Cte«©^W©=fcd^^©i|#M^©tiiB : Sr®^LT*i< Ifcf, *^7*t ■J/L'->3 > 
ffl2fScJ&»3pWCS*o :nciil'«Djtt — «fc LT3tlR(15)( TAn Efficient and Accurate Camera 

Calibration Technique for 3D Machine VisionJ .Roger Y. Tsai.Proc. CVPR'86. pp. 364 — 374, 1986) f-IB<? 

»<**iS©S»*i»ttt U-C35^l(caffi*lS3eU. fe&?^tK* 5 diT'&9f[ii©35 #CJ:fi3fcB®aflSK:.J:oT£i; 
■CV^44b<*«iS©S -^(^J^X feM^lni N(U 7 V) 45.flgw<o§6js ?t\ Life 

KU J(t*,v) = B(u,v) max(y^ (ifZ* • J^(t*,t;)),0) 

— * 

[ * 2 7(«, «) = ») • £ (£ = 53 ULi) 



^IfjiB&gf© ^-Hfifiti, ©*HR»fc&a;63fc«©t>i:-c©Bfci: U ft *!!©;£ fa* A 2) iB^©Si?B#©j& 
©TTWS^liifi— ©Mil ©SHfe©fa-ejoHt?>©"C, *-i^TOilftO!)*Tt»"CJb4. J&iJH^IfrBte g¥© 

8K(14) ( ropenGL^o ^7 § ^{fii-i KJ .Mason Woo. Jackie Neider. Tom Davis. Tv? V > • -> i ^ W-f 



_ _ . . _. . Itm&Zfr 

Bftp®Bft^ iNTffl©*:*, ^-afc*-** ^*(*" = lj 2, • » • , N)-e&z,fr-rt^ V 

K3] 



S=[K 1 K 2 ... l^ivj 



v = —ss T 

N 

K4]_M 

^ i==1 ^ x 100 



— #• 

\& ■ r 

— *■ 

K5] n 



1=1 



™D = \f q - I c \ 2 
D < D f 

[gin \t*&w(Dm&M&&W(»n\(DnMm(»fam<Dffin*^t -7vv?m. i®2] \*m&®x*&z> 0 

3^7C^t>tSiJ^g:llO-e{i, 3»:7C^alJ^^e : S:fflV^Tt)#:cD3*7C^tt ; S:jSlJ 5£-T5o fll xtf OI#®m 1-123687 

sw^iffl^#si2o-efi, '3-&7cMVu~*tfc-t %ya&v)WiW(D&.mm*wmT 5„ ^j^t^ttrsa c^sh^fi 1-123687) 

o 

(R x ,Ry, Rz\ -jjtvvfa. &mm f, mmn CXttm^Zo ^n&o^y^&jpjjB*** 

«W*|jE#«230Ttt, WrlEffi«Sf»«t^S220CD|g*4:WfflLT, A#H& t W Cffi«K»K:*o T*«biftV% 

0° A>P,9 0° 36ia«©M 

So 

■ 

BJije^HSEo^&AfcsnsftBSi^^ flaw*#«'(b#S23i*»6A*sn«ji8i»*#»K:*5»iiia 

^tLTtfv*-**^?*^, feMBcS^f^fcLtTci ^SL^r-^ufc/iH^cifc^-etSo :nf> 
V>iES«^^(Z)i:-rS3i:T, Steffi * fa ± $ 1+ 4 3t*<-Cgr*. 3 coili«i±^ti a y tf a - ^ 7 -r 



MWIE»ffiM*feia*S233-ett, H«£Ji!l#»232T£j£SnfclltlW£|bB«# *£>JR9i&H&M& [*3] K L 

\jj - 

H*«©*tv^»r Af mamm** h lt^^s, d ©isms'* * wu©& Af %- 

«»ii:LT [*4i -etuis ft*** *-**a*85?<*ja*.5»i: LTifeje-rscii, raw^i;@^©iii^©f!c^ 

^M^i > 0.95 „_ 

* . 

ia«jttt¥«24o-ett, A*iii*4:4j«snfcifcttis*©«<KflEowia5flt*w- m-r-So mmmcDiinjj&\zi$m 

*t£&ffi&mmx%Z>TbK ■*©— - WtUTfi. K6] ©J:-5K:iHflfc©#ia*<Z>WflE«©36©Zl*Sl3&:4f*fllV^ 
53i:4«T?t4, C©ftfcf3, S3K(12)( ra V fcTa C«fc S|R©B»— 9— ^-f-J . tfl *iHt**lfc*:tt 
D-II. Vol. J80-D-II. No. 8. pp. 2031—2046. 1997) Cjff L < 26^ 5>ftTV>.5 J: d a. fciPJJB RrtBT?** 

o 

mtmmznxi^m&iz^ swe23o*»& 2 4o <Di&m*n^xnffim*ft%-i, *r©4M*fc**>«Tv>sa»0 

[06] tt^ftiHOBfeJIg^M©^©?*;^©^©*^^? yn 7 ^^5„ #HJE#Ki, SHl©S£ _ 
if, S«#«2100K:*5V'»T, ^©JRlfrfcJIV^SiRT'-^fcLT* <&>{*:© 3 &7c?BM*t, «&©jlf8&#© 

3*7c»ttji!isg#«2iioTtt, m\(ommm(D3}!k7tM^mi^.^miotmm^ mm&i 1-123687)03^^^111^ 

^****H»*i*¥R212(n?tt, ltfEt(tt®*ii«©J*l0§*#£*b^«!fr& aasn*MiH*fl=fcRI*&JHW 
*#**BC»j£l,-Cftfl:©Bfc*JM*U f^^^Bfti: LTffiJtl,, «Atf*fM&:&C*fl:«:<t l 'C>i: t 

] i^snsM (^j 0)c*5V<>-c, @i<t>iz~o^T — -90«*»e> 90s*-e©isiB-e^n^n 15s 
*fc:fri*:&<spjjB«rte -eafei>o «rE©#ftcj:9»»L;fcB«»*^*>i* + B^fc LTma-rs. 

ja±©a»«2100ffl»S*fTofc4&#fc:JttL-C, J»TF©JI8*#»2200C:J3V*T Bi££fflV>fc^#:J&£-#LS ; §:^ 

a«#«22io, ffie^^«s*is222o«. ^n^nBuiemifflHss^jicfcit^. m&^mio, &.m^mmm^m2 

*mmm<DmwffiiE^m2oo\t. mmm\e>mmmizi3rtz>MW3kwmt^&. 231*^ < . H«£jiS#«2232K;fc 
v^t, mur? z**m&mm^m\2Mz *>\^xm&Ltzr ? vmmmz^wssva&mMvimmmizLx 
m\^z £XMw&wjmmm*§i)&Tz>£tfmts;-*x^z>o mi] ti** *«c*iit5H8MfmE^«©atn* 

■ 

B»£JBK^«2232T?tt, f-^M*«21304»&J»**rftTJ**te{* J©3#c 7tJEM*fc fj/ **- + Hft#*K* 

a*. ffi«s?»«3e#a222o*»&-5-A.p>nfc tt«ss»t, wse^***-*B«B©#y*;**-*B**fflv>-t\ 

*io, x^>t.^-viS^(i?lll«rig^$nfciSi^-cab?.©-e, ^1©H tfl«)J:^i:3>t'i-^77<^ 
^©Sff t i o -c i> ») *ii*t 4!6&Kiifcv\, 



ikiz, *^(W©m3^D^^E^J^rov^T8¥*IH^rS^W-r* 0 ^MMfflit mim.ll tE«©ftWfc«a-r*. 

»*t«i5e¥S3iio casv^-c^-c 6D*ft03&5c?EM*%iii5£-raj&»to»)fc:» io-&^L^»©3*s?t*tt«:fHB.rr 

SfittTr, ¥W^^*S3150C*5V^T^lo7 t cC v> L'>»©3^7C^©¥*^ i: &-53i*7EJ!M*l o£fcB#L, 
m&ftMLtKZ-kxwyafcoyBftmifflLtji *5=fci>\ !^^■^m3200^ci5V^Ti^ieW©3»:7t:^=&flJffl 

e«*«3100CfeV^X, feftWi^K/BV^a^T-"--? £ LT, feftl fc!$lft:2©2o©!|gjft:i;:ol>T, 3 
JKTcJ&tti:, *ft©JRW*ff:CD«,i:-C ©Hfer-*-* LTfc < . 

3»C7CJEJttflJ^#S3110-Ctt, OKfBm 1-123687) ©SifcTGjIWfllJeS&B&fflUT feftli:%#2©3 JfcjcJIMfc&ffll 

TOJ§HttfflJ£^8!3l50Tii, [09] ©fiEHl^-T 2o© »#03»C7C»tt*M.C?*— J: -5 Cf 

^¥ibL, £H© 2fHi(r SKfcWrffi*a*!i£MPH-CttJeu ^^©BftB^T^TEMK&ff-g-f £> 0 [09] 

(<bi,»i,#l). (#2,2/2,22) 

ai+xa 3/1+3/2 Zl±£2\ 

"T4. 



( 



s*r*iii«¥«3i2o-ett, *fri.*#2-*n-^n03jfc7c»ttc*r«t-« ^©SificDKW'feWJt'r*. «ris( 

4WP1 1-123687) ©3Jfc7cJ§H*SHeStB *JBVvh.», 3*5c»«fcRI«Hi*ffiCDfet*«*Slse-r*^ ta 1 ?**. 
J2HT TJiiOfeflMIfcKlf^Cft/l-rs. 3 pJ&»«<545-j«i: - d.TJfllJSLfc 4Mbfr©fe**eD*fiStt, &©£ 

5fcLTSM£-CSa. -f&fc>*>, ¥i£#l*"C rra©3 *7£tft^ti-*TaRCfllv^*fl:u *#2©3 

f'-^Ett^fMlSO-ett, 3&ro#lftffl£#«3110.*9m#Stt*lltrE 3 PiW& ttfc, Sftrssjgij^^g:3120<i:Om 

jjn5fefti,ft^2©^n^«^ Etrra. 

JIR£-fM9!3200C:l3l***UH3\ »*l.*#20Jg«4: LT 7-*-*i2tffM£3130 A»6R*tB;Sn*3JJC7EJ|Jtfta<»E 
W»tKT*S£©*##l©*;afi«i:#*fc 0, ftfeomaii^:TI^1«-Cab*o 

«_h, ***«Ttt2ffl©»fr*a»rSIRC^OW^tt«rffi«-t**IS« tttWLfc;*, cniife<*n^j 
iCiDH |gTab£„ 

JKC #»«©*4©flil^K:ov^SMiBCISiW-ra» #f£#£Wi tff*^30 nE*©l£e.!c;fg^-f -5 0 *%;8t0y 
[Bio] tt^aMosftfls^-aao^osMfiwoftaostnsrwf ?m-z%>z> a **Mm\t. fsicomm 

L WkMfoMk. LTa37-r*«r«jftti««ia#fa!4140a«jainSiiTV^SA, x-?E1fi#©4130C;fcV>T^ft 

0#g^ott«tE«rsj«, fcii/, ^m2oo©fce£§§j£5e#i8!4220K*^T, T'-^Ea^miso 

S?\ SWSttlOOCJ^T, ftftCi^CfflViSlSf-^ i: LT, fefr©-3 «7C»tt, RW**WJeU ft 
3*7G»#;fc«fct/R#*fr&4fe#©4#*j6©3 JfcTctffll***, ME3 #C7t7EM*i:, KM^, ttfft£ftB*fiSL 

3*5c»«i«je^«4110-ett, 3&^t*i^Sg*ffl^T^#©3&^tt£iflJ JS-ra. ***«T?tt3»:7C»tt 
«£^g©~0!it LT(iftgH¥l 1-123687) ©3 *7cJIM*illlSE«B*fliV'»Sa*, £ ©ftfe£ frsgB^JOTtlT 

3b o 



©&K£&aiL, -^©S^Tca?^ «r«t,«ffilt LTUJAtS. — «fc LTA»Oli«:J»#*fft** 

LTv^auffii^ars. inu#ii-cff^:fc ^t-^^l, sftwcff-^^Si: LT«rExitt (2) (#^28 

7.2776-S- rgSOi «&R3^^ffiJ )^f, A«t(5)(#P>B s F6-168317 ffliAMfcSiJttBJ ) E«©^j*«:4f«« Jfe^ifefcipJ 



m -et&o *s6ifi«yc»i [012] tr^T£?&fee©i2<@©,6(o~ii)ft# WMtLxm\^ a :nb®m« 



Ai = (xi,yi,Zi) t (i = 0,1,— ,11) 



■9o 



& % ffl v > % ft 



ftJtS-T-5. [011] E^«4220©8ffl%MWT*'7nv*IHft3R-r. 

«f«jStta*a422i-ett, irea«#ia4iooK:ijit44#*jattffi#ft4i40-c«i fflLfe»5f«jft» ^»4:hi:**» 

•^i = (Wtjfi)j = 1? ^5 * * * J ^)*A*i«#iAfiIi: LTiij^-rSo cnii 

168317 xmAwammi ) tc*©^*3te^roE«iR^«iffl^«4i4OT?fflv>3fe©fcH«0«i*<c*j**w ffl^rtg 

ffi«S?»fi-*^«4222-ett, flMEA#iBfc4#*j6ffi«fc, t*-*BK#«4130 fr6K*liltW|j«ttt*flfl!L 

(imfta^ (15) ©#&fr^*&#?fe& 5 ?yffl^t&T&£#, *SH6«IT?tt-«i:LXtfc »»<5^5^^J:L 



ftLAEKft »fgi0^7^ * fc-g-«>T&«»»fctf.s:o Aizxti mm#9utLVL 



B 



t©M 



«7] 




= / 
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W8]T ~ _ _ _ _ 

a 
6 

c 

K9] ^$ft£®teft^-rfr?U-e£>5o 

[5$9] 

cos fl y cos R z — cos iij, sin R z + sin sin Ry cos J2 Z sin J? z sin Rj. + cos i?x sin Ry cos J2 Z 
■R = cos J? y sin J2 S cos R* cos 4- sin sin iJj, sin R z — sin /£„ cos R z -4- cos itr sin Ry sin R z 
— siniJj, sin .ft, cos Jl y cos i? x cos -Rj, 

^ -Ry, T!j. , Ty , , / ^ aa^^t J; 0**4. £©*3fl:0ln-*£tt«*&# 



fled, *£i«©l|S5©*;fc«CoV^P*IICt5li»-r*o *HS£#Jt4 3f#iS28 fciu? »3£Jg29 fcaa«©»9IKffi 

at- a. 

®j&«©{t#tff r ff>rviHK:j:oT«w>&n-c v^sfc-rs. *iwtji4, *i©5ms«*:j*«u sacTcjBttsisg 

3*7cJf?«jW»S!ll0tt, KttHOCADf*-**, RR-fr^SflOO-eajtaT*-* LT3&7cftMK 1 L X 

RS^ A*)©*©**** if ©JEflJK 

W-T+^T* 3&^ttMM*iBJtettr»BW*£«#«J0lT!S.5. Sfc, 3* 7cjfEM£A ? g3§$iaT 

1>5©T% A*S«lC*tt**ftffl3»C7C»4:tt«»»© ^»C*t LT5c£KMIE-t-5 ^ t^Xt £ 0 JH 
WMfflSfl^LTt +fl-fc*|jE4?T-5 *%H©JR9iltjEftUlli, *iML®#.<?)mm<D 

ttd* 3E£tfcSLffiT?*^fc©Eol*-C ^jgfflT*t> ^#©flfc©S|5#{rJ:5}£i!I©3gi8c T-eSS^X^OfeifK: 
*fUTl)MfflX§5o Lfti«it*tt(8).(9)J:0 fcKiKV^H©— ftftjfcfcfrfflBa-frfcSJBTS 3 fctfT? 

*fc, JHwsftia«w*a«sl^■cv^s3»c76Jl^tt^:s#**fflv^Ti»»t aat&s 



[01] tt, *AW©ISM*S1E©&l©X;Ktt©A9©flEn ?:St/D^g], 

[02] t±, *&w<Dm®.m& , &s.(Dm\<DmMm<Dffif&m. 

[03] «, #j£^©gii©ii;^y©M^li:E&s©ms©^ft&^-r /n^^ 

[04] 14, JRW*#*fcA*Mifl©*#C#-r5#|fiJ*«-rftK £l89!-r50, 

[05] tt, H«*fUfflUfc4b#JH-&«E*©«fewUfcH, 

E06] »4, #»ili©*2©**W©M©i!llft&ij*-r7n v *EU 

[07] f4, *5IM©*2©*aSW©RRM*f ! l : IBiE*ia©a[liS:^-r ?d^bi, 

[08] fi, *^^©^3©n^j©^a©sfin : S:^-r-/n „ ^h, 

[09] f4, W»«©±Jrt^Jfe*KiB-r*H, 

[010] »4, *»M©»4©*JE«©fta©«En«:^-r^n?;^BI, 

[0ii] (4, 3|c»w©»4©5gififfli©ffi«*»j(t3eaa©iitn 5 S:3sr 

[012] B\ *MR»#©*MRAi: LTffll>-5feft©S15ft©-#J£^T0, 

[013] tt, t£*©il«lJ&-£-i£#r©-0iJi: Lt, aSH^^JH^Tt tC2Jfc7C©Bife©*S:fflV^4a«©1llj««: 
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Title of Document 

Specification 

Title of Invention 

DEVICE, METHOD AND RECORD MEDIUM FOR IMAGE 
COMPARISON 

Scope of Claim for Patent 

1. An image comparison method wherein : 

in a registration step for registering data of each object to be 
compared in a database, three-dimensional shape of the object and 
surface reflectance of the object are measured and stored; 

in a comparison step for comparing an input image with the 
registered data, a two-dimensional image is photographed as an input 
image by use of video camera, etc., illumination variation images being 
images of an object in predetermined certain position/pose under various 
illumination conditions is generated by use of the registered 
three-dimensional shape and reflectance data, an image being most 
similar to the input image is obtained from the illumination variation 
images to be generated as a reference image, and an evaluation value 
concerning similarity between the reference image and the input image 
is calculated, and such processes are performed as a verification of 
whether or not the reference image is the target object, a search for a 
registered object corresponds to the target object, and a search for a 
registered object which is similar to the target object. 

2. The image comparison method claimed in claim 1, wherein, in the 
reference image generation step, an illumination variation space being 
an image space which is spanned by the illumination variation images is 
generated, and an image which is most similar to the input image in the 
illumination variation space is generated as a reference image. 
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3. The image comparison method claimed in claim 2, wherein: 

in the illumination variation space generation step, illumination 
variation space is generated by obtaining the basis vectors of a space 
that almost accommodates image variation due to the illumination 
5 variation by conducting a statistic analysis such as principal component 
analysis to the illumination variation images? and 

in the reference image generation step, correlations are obtained 
between the basis vectors and the input image , and the reference image 
that is the most similar to the input image is generated by use of the 
10 basis vectors and based on the correlations. 

4. The image comparison method claimed in claim 1, wherein: 

in the registration step, the illumination variation images are 
generated beforehand, and the illumination variation images are stored; 
15 and 

in the comparison step, the registered illumination variation images 
are used for generating the reference image. 

5. The image comparison method claimed in claim 2, wherein, 
20 generation of illumination variation images and an illumination 

variation space in the comparison step is preliminarily performed in the 
comparison step, the generated illumination variation space is stored in 
the registration step, and the registered illumination variation space is 
used for generating the reference image. 

25 

6. The image comparison method claimed in claim 3, wherein, 
generation of illumination variation images and basis vectors in the 
comparison step is preliminarily performed in the registration step, and 
the generated basis vectors are stored, and the registered basis vectors 

30 are used for generating the reference image in the comparison step. 
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7. The image comparison method claimed in one of claims 1 to 6, 
wherein position/pose parameters of the object of the input image are 
separately inputted instead of employing the predetermined certain 

5 position/pose. 

8. The image comparison method claimed in one of claims 1 to 6, 
wherein the position/pose of the object of the input image is estimated by 
use of the registered three-dimensional shape and reflectance instead of 

10 employing the predetermined certain position/pose. 

9. The image comparison method claimed in one of claims 1 to 8, 
wherein, instead of measuring the reflectance, image information is 
photographed under an adequate illumination to be used as a substitute 

15 for the reflectance in the following processes. 

10. The image comparison method claimed in one of claims 1 to 8, 
wherein, in the registration step, the reflectance is measured in such a 
way that image information under an adequate number of illumination 

20 conditions is photographed as texture images, and in the comparison 
step, for generating the illumination variation images, the texture 
images are employed instead of changing the illumination conditions. 

11. The image comparison method claimed in one of claims 1 to 10, 
25 wherein^ in the registration step, instead of measuring 

three-dimensional shapes of plurality of comparison target objects, by 
measuring one or a few number of three-dimensional shapes, one 
three-dimensional shape being the one three-dimensional shape or the 
average of a few number of the three-dimensional shapes is outputted, 
30 and thereby, shapes of all target objects are not measured; and 
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measurement of the reflectance is performed in such a way that the 
reflectance of all comparison target objects is measured or image 
information is photographed under one or a plurality of illumination 
conditions, and only the average shape data as three-dimensional data in 
5 the following processes. 

12. The image comparison method claimed in one of claims 1 to 11, 
wherein the three-dimensional shape is not measured and 
three-dimensional shape is inputted separately from drawings or the 

10 like. 

13. The image comparison method claimed in one of claims 1 to 12, 
wherein the reflectance is not measured and reflectance and color 
information are inputted separately from drawings or the like. 

15 

14. The image comparison method claimed in one of claims 1 to 13, 
wherein, in the registration step, the comparison target object is 
registered with position information of a feature point. 

20 15. The image comparison method claimed in one of claims 1 to 14, 
wherein the comparison target object is an automobile. 

16. The image comparison method claimed in one of claims 1 to 14, 
wherein the comparison target object is a human face. 

25 

17. An image comparison device wherein^ 

a registration means for registering data of each object to be 
compared in a database comprises; 

a three-dimensional shape measurement means which measures 
30 three-dimensional shape of one or a plurality of comparison target 
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objects to be registered in a database, and outputs the three-dimensional 
shape of the objects; 

a reflectance measurement means which measures reflectance of 
surface of each position in three-dimensional shape and outputs the 
5 reflectance as reflectance of the object; 

a data storage means which stores the three-dimensional data and 
the reflectance as registered data, and reads out and outputs the data for 
a comparison process when necessary; and 

a comparison means for comparing an input image with the 
10 registered data comprises- 

a photographing means such as a camera which photographs and 
outputs an input image; 

an illumination condition variation means which sets various 
illumination conditions and outputs them as illumination conditions; 
15 an image generation means, by being inputted a predetermined 

certain position/pose and three-dimensional shape aid reflectance 
outputted from the data storage means, generates illumination variation 
images in the same position/pose as the position/pose under the 
illumination conditions; 
20 an illumination condition estimation means which obtains an image 

being most similar to the input image from the illumination variation 
images and outputs it as a reference image; 

an image comparison means which calculates an evaluation value 
concerning similarity between the input image and the reference image 
25 and outputs the evaluation value; 

a comparison judgment means which receives the evaluation value, 
and performs such processes as a verification of whether or not the 
reference image is the target object, a search for a registered object 
which corresponds to the target object, and a search for a registered 
30 object which is similar to the target object, and outputs the judgment 
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result. 

18. The image comparison device claimed in claim 17, wherein^ 

the comparison means includes illumination variation space 
5 generation means for generating illumination variation space which is 
spanned by the illumination variation images and outputting the 
illumination variation space; and 

the illumination condition estimation means includes an 
illumination condition estimation means for generating an image which 
10 is most similar to the input image within the illumination variation 
space as a reference image. 

19. The image comparison device claimed in claim 18, wherein* 

the illumination variation space generation means generates 
15 illumination variation space by obtaining bases vectors of a space that 
almost accommodates image variation due to the illumination variation 
by conducting a statistic analysis such as principal component analysis 
to the illumination variation images! and 

the illumination condition estimation means obtains correlations 
20 between the basis vectors and the input image, and generates a reference 
image that is the most similar to the input image by use of the basis 
vectors and based on the correlations. 

20. The image comparison device claimed in claim 17, wherein: 

25 an illumination correction means includes illumination variation 

space generation means which generates illumination variation space by 
obtaining bases vectors of a space that almost accommodates image 
variation due to the illumination variation by conducting a statistic 
analysis such as principal component analysis to the illumination 

30 variation images! and 
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the illumination condition estimation means obtains correlations 
between the basis vectors and an input image, and generates a reference 
image that is the most similar to the input image by use of the basis 
vectors and based on the correlations. 

5 

21. The image comparison device claimed in claim 17, wherein: 

the comparison means does not include an illumination variation 

means or an image generation means; 

the illumination condition variation means and an image generation 
10 means are included in a registration means; 

the storage means stores the illumination variation images; and 
the illumination condition estimation means in the comparison 

means uses the registered illumination variation images. 



15 22. The image comparison device claimed in claim 18, wherein: 

the comparison means does not include an illumination condition 
variation means, an image generation means or an illumination 
variation space generation means; and 

the registration means includes the above three means- 
20 the storage means stores the illumination variation space; and 

the illumination condition estimation means in the comparison 
means uses the registered illumination variation space. 



23. the image comparison device claimed in claim 19, wherein: 
25 the comparison means does not have an illumination condition 

variation means, an image generation means or an illumination 
variation space generation means; 

the registration means includes the above three means; 
the storage means stores the basis vectors; and 
30 the illumination condition estimation means in the comparison 
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means uses the registered basis vectors. 

24. The image comparison device claimed in one of claims 17 to 6, 
wherein^ 

5 a position/pose input means which separately inputs; position/pose 

parameters of an object in the input image instead of employing the 
predetermined certain position/pose is added; and 

the illumination correction means uses the input position/pose. 

10 25. The image comparison device claimed in one of claims 17 to 6, 
wherein: 

a position/pose estimation means which receives the input image 
outputted from the photographing means, and estimates the 
position/pose of the object in the image, and subsequently outputs the 
15 position/pose is added; and 

the illumination correction means uses the estimated position/pose. 

26. The image comparison device claimed in one of claims 17 to 24, 
wherein a reflectance measurement means in the registration means is 
20 an image photographing means which obtains image information by 
setting adequate illumination and outputs the image information as a 
reflectance, and a plurality of the image information is used as a 
subsequent of the reflectance. 

25 27. The image comparison device claimed in one of claims 17 to 24, 
wherein the reflectance measurement means in the registration means is 
an image photographing means which obtains image information by 
setting adequate number of illumination conditions, and outputs a 
plurality of the obtained image information as texture images! and 

30 there is no illumination condition variation means in a comparison 
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step; and 

the image generation means generates images by use of the texture 
images instead of varying illumination conditions. 

5 28. The image comparison device claimed in one of claims 17 to 24, 
wherein the registration means includes an average shape generation 
means which receives inputted one or a few three-dimensional shapes, 
and outputs the three-dimensional shape or one average shape of the few 
three-dimensional shapes; 
10 the data storage means stores only the generated average shape; and 

the comparison means uses the average shape. 

29. The image comparison device claimed in one of claims 17 to 27, 
wherein there is no three-dimensional measurement means and the 

15 three-dimensional shape is separately inputted from drawings or the 
like. 

30. The image comparison device claimed in one of claims 17 to 25 and 
11 to 12, wherein there is no reflectance measurement means and the 

20 reflectance is inputted separately from drawings or the like. 

31. The image comparison device claimed in one of claims 17 to 27, 
wherein: 

the registration means further includes feature point extraction 
25 means for adding, in the registration step, to the registered data, feature 
point position information which is information about positions having 
feature appearance of the comparison target object as a necessary 
information for estimating the position/pose. 

30 32. In the image comparison device claimed in one of claims 17 to 30, 
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an automobile image comparison device wherein the comparison target 
object is an automobile. 

33. In the image comparison device claimed in one of claims 17 to 30, a 
5 human face image comparison device wherein the comparison target 

object is a human face. 

34. The image comparison device claimed in one of claims 17 to 32, 
wherein, as a photographing means for obtaining an input image, an 

10 image input device is employed for reading films, photos, documents and 
the like. 

I do not describe claims for a record medium, I would appreciate it if 
you could compensate for them, thank you. 

15 

Detailed Description of the Invention 
Field of the Invention 

The present invention relates to a comparison technique of objects, 
and in particular, to an image comparison device, an image comparison 

20 method and record medium which makes a data processing device 
execute the image comparison, in which three-dimensional shape, 
surface reflectance, color information and the like are preliminarily 
registered, it is possible to conduct comparison correctly withstanding 
variations of shooting conditions such as position/pose of the target 

25 object in the image and illumination conditions. 

Prior Art 

As shown in Fig.5, image comparison techniques are such techniques 
that an object existing in the three-dimensional space in a random 
30 position/pose is photographed to acquire one or more input images by use 
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of a photographing device such as camera or video camera. Thereafter, 
the input images of the target object are compared with images of objects 
which have preliminarily been registered. The image comparison 
process is generally composed of two steps^ a registration step for 
5 registering and storing the compared objects; and a comparison step for 
comparing the input image of the target object with the registered data 
of the registered objects and thereby judging what the target object is. 
In each step, the photographed image is used as a two-dimensional 
image without being processed, or is used after being converted to a 
10 three-dimensional shape by means of the technique disclosed in 
Document 1- "3D image measurement", Iguchi, Sato, SHOKODO. In the 
following, conventional image comparison techniques will be explained in 
detail with reference to documents. 

15 As an example of an image comparison technique in which 

two-dimensional images are registered beforehand and a 
two-dimensional image is used as an input image, there is disclosed in 
Document 2: "face image comparison device" in Japanese Patent 
No.2872776. This prior art has been designed to conduct image 

20 comparison of human faces and has such configuration as shown in Figs. 
1 and 3. In the registration step, two-dimensional images are obtained 
by use of a camera 11, and the obtained two-dimensional images are 
stored in a storage means 12. In the comparison step, a 
two-dimensional face image is obtained as the input image by use of a 

25 camera 13. A normalization means 14 extracts feature points being 
eyes, nose, etc. to be used as the criteria for judging pose, size, etc. from 
the input image by means of image processing. A normalized image 
which is normalized in accordance with two-dimensional position and 
size on an image is outputted by use of the feature points as coordinates 

30 position. Thereafter, an image comparison means 15 compares 
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registered images reads out from the storage means 12 with the 
normalized image by means of pattern recognition, and outputs the 
result of the comparison. 

As an example of a conventional comparison technique using 
three-dimensional shapes, there is disclosed in Document 3* a "personal 
identification device" in Japanese Patent Application Laid-Open 
No.HEI9-259271. In this prior art, there is configured as shown in Fig. 
14. In the registration step, a three-dimensional shape color 
measurement means 21 measures three-dimensional shape and color of 
each comparison target object, and the three-dimensional shape 
information and the color information are stored in a storage means 22. 
In the comparison step, a three-dimensional shape color measurement 
means 23 measures three-dimensional shape and color of a target object 
as input data. A translation/rotation means 24 translates the input 
data so that the barycenter of the input data will be on the barycenter of 
the registered data. Subsequently, the translation/rotation section 24 
rotates the translated input data in various ways, and a minimum error 
calculation means 25 obtains the error between the rotated input data 
and the registered data. The minimum error calculation means 25 
adjusts the three-dimensional position and pose of the input data. 
Thereafter, the minimum error calculation section 25 compares the 
corrected data with the registered data. 

A "personal identification device" disclosed in Document 5- Japanese 
Patent Application Laid-Open No.HEI6-168317 uses two-dimensional 
images both in the registration step and the comparison step, and the 
configuration thereof is as shown in Fig. 15. In the registration step, 
two-dimensional images are obtained by use of a camera 41. A feature 
extraction means 42 extracts feature points where variations in pixel 
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intensity are large and stores position data of the feature points in a 
storage means 43. In the comparison step, a two-dimensional image is 
obtained as an input image by use of a camera 44. Subsequently, a 
feature extraction section 45 extracts feature points where variations in 
5 pixel intensity are large from the two-dimensional input image and 
outputs position data of the feature points. Subsequently, the 
comparison means 47 conducts the personal identification by comparing 
the registered feature point position and the feature position of the input 
image. Here, in order to compensate for and absorb variations in 
10 position/pose of the target object, the feature point position data 
outputted by the feature extraction means 45 is normalized by a 
position/pose normalization means 46 by use of a three-dimensional 
shape model of a standard object which has been prepared in the 
position/pose normalization means 46. 

15 

A technique disclosed in a Document 6: Hiroshi Murase and Shree K. 
Nayer "Visual Learning and Recognition of three-dimensional objects 
from Appearance", Int. J. Computer Vision, vol.14, pp. 5-24 (1995) uses 
only two-dimensional images both in the registration step and the 

20 comparison step and executes correction not only for variations in 
position/pose but also for variations due to illumination conditions. Fig. 
16 shows the configuration of the conventional art. In the registration 
step, a large set of sample images of each object are obtained by a 
photographing means 71 by automatically varying pose of the object and 

25 illumination conditions so that all feasible poses and illumination 
conditions can be covered. A manifold calculation means 72 obtains 
basis images capable of expressing the variations of the sample images, 
by means of principal component analysis. Subsequently, the manifold 
calculation section 72 generates a feature space which is spanned by the 

30 basis images, obtains a trajectory of the sample images in the feature 
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space as a manifold, and stores the obtained manifold in a storage 
section 73. In the comparison step, a two-dimensional image of a target 
object is obtained by a camera 74 as an input image. A distance 
calculation means 75 calculates the distance between the input image 
5 and the manifold in the feature space and executes the comparison using 
the distance as a yardstick. Thereby the comparison by use of the input 
image, which can be obtained under various illumination conditions and 
poses, is made possible. 

10 Changes occurring to a two-dimensional image due to variations of 

illumination conditions when the position/pose of the object is fixed has 
been analyzed in detail in a Document 7" Peter N. Belhumeur and David 
J. Kriegman "What is the Set of Images of an Object Under All Possible 
Illumination Conditions ?", Int. J. Computer Vision, vol.28, pp. 245*260 

15 (1998). In the case where the position and pose of the object are fixed, 
an image obtained under an arbitrary illumination condition can be 
resolved into images each of which is obtained under a point source of 
light. Therefore, an image obtained under an arbitrary number of point 
sources of light can be expressed as a linear combination of images each 

20 of which is obtained under a point source of light, in which the intensity 
of each point source is used as each coefficient of the linear combination. 
Based on the above analysis, a method called "illumination subspace 
method" which is shown in Fig. 17 has been proposed. 

25 Based on the above analysis, in the Document 7, there is suggested a 

method structured as shown in Fig. 17, called "Illumination Subspace 
Method (hereinafter referred to as method (8))". A photographing 
means 51 selects and sets three or more illumination conditions so that 
the number of pixels in shadow will be as small as possible, and obtains 

30 images under the three or more illumination conditions. A normal line 
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calculation means 52 obtains reflectance vectors on the surface of the 
object, with regard to each pixel in the images, by means of principal 
component analysis. An image generation means 53 successively 
generates images which are called "extreme rays" and stores the 
5 generated extreme rays in a storage means 54. In the comparison step, 
a two-dimensional image of a target object is obtained by use of a camera 
55 as an input image. When the reflectance property of the surface of 
an object is perfect scattering and the surface is convex, an image of the 
object under an arbitrary illumination condition can be expressed by a 

10 linear combination of the extreme rays with positive coefficients, and 
therefore, an illumination correction means 56 generates a reference 
image being an image of the registered object under the same 
illumination condition as in the input image by use of a linear 
combination of the extreme rays. The coefficients of the linear 

15 combination are determined by use of the nonnegative least-squares 
method. An image comparison means 57 compares the input image 
with the reference image by calculating the similarity between the two 
images. 

20 In a technique disclosed in a Document 10" A. S. Georghiades et al. 

"Illumination Cones for Recognition Under Variable Lighting: Faces", 
Proc. IEEE Int. Conf. CVPR, pp. 52-58 (1998), when the extreme rays are 
obtained according to the method (8), pixels that should be in shadow are 
found out based on the three-dimensional shape of the object by use of 

25 techniques of computer graphics such as ray tracing, and the pixels are 
shadowed. The technique of this document aims to apply the method (8) 
to objects having concavities. 

The aforementioned Document 7 also proposes a Sampling Method 
30 (hereinafter referred to as method (9)), which is shown in Fig. 18. It 
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takes much time to calculate all the extreme rays as in the method (8), 
and therefore, a photographing means 61 in the registration step selects 
and sets an appropriate number of illumination directions so that the 
angles ( 9 , <f> ) shown in Fig. 4 will cover the sphere at almost even 
5 intervals for example, and obtains images. The images obtained as 
above are substituted for the extreme rays. Thereafter, in the same way 
as the method (8), the illumination correction is executed by means of 
the nonnegative least-squares method and thereafter the object 
recognition is executed. 

10 

Problems that the Invention is to Solve 

The target object in front of a photographing device being camera, 
video camera, etc., generally moves in the three-dimensional space 
(parallel translation, rotation, etc.) unless the position/pose of the target 

15 object is fixed or adjusted. Further, the illumination conditions vary 
every moment when the photographing of the target object is done 
outdoors. Therefore, the appearance of the two-dimensional input 
image of the target object is necessitated to vary widely. The 
conventional techniques could not compensate for and absorb enough the 

20 variations in the position/pose of the target object and the illumination 
conditions, thereby the application of the conventional image comparison 
techniques used to be confined within narrow limits. In the following, 
the problems in the conventional techniques of the above documents will 
be described in detail. 

25 

The comparison technique as disclosed in Document 2 employing the 
simple image comparison between two-dimensional images can not cope 
with variations of appearance in two-dimensional images caused by 
changes of the pose of the target object being three-dimensional rotation, 
30 etc., illumination conditions, etc. Therefore, the application of the 
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technique of the Document 2 is extremely limited. 

The image comparison technique disclosed in Document 3 requires 
three-dimensional shapes not only in the registration step but also in the 
5 comparison step. A three-dimensional shape measurement device 
described in Document 1 becomes necessary on each comparison, and 
thus, the cost for the comparison is necessitated to be high, especially 
when the measurement of the target object has to be done away from the 
place where the registration step has performed. Further, for the 
10 measurement of the three-dimensional shape, each object has to stand 
still until the measurement is completed, and a darkroom etc. becomes 
necessary for obtaining precise data. Therefore, the application of the 
technique of the Document 3 is also limited. 

15 The technique employed in the Document 5, which detects pixel 

positions where variations in intensity are large, is effective for 
comparing objects such as blocks having large three-dimensional 
curvatures, black marks on a white board having steep change of 
reflectance, etc. However, the technique is not suitable for comparison 

20 of human faces as mentioned in the Document 5, which means that 
stable and reliable detection of the feature points is difficult. The 
Document 5 also refers to a correction of pose by use of a standard 
three-dimensional shape of target objects, however, the method can not 
be employed unless the objects are similar in shape. 

25 

In the technique disclosed in the Document 6, enormous amount of 
two-dimensional sample images have to be gathered, taking the 
possibilities of various illumination conditions (two or more point sources, 
extra lights other than point sources, etc.) into consideration. Further, 
30 the shape of the manifold in the feature space is not defined at all, 
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therefore, it is difficult to find appropriate sets of parameters when the 
distance between the input image and the manifold is calculated in the 
feature space. Therefore, large amounts of calculations are necessary. 

5 The methods (8) and (9) require images of the registered object 

illuminated from many directions. A specially designed lighting unit 
has to be used in the registration step, and precise netting of the 
illumination conditions is difficult from the viewpoints of placement and 
setting of equipment. Further, in the method (8) and (9), if the position 

10 or pose of an object changed, images of the object in the new pose have to 
be obtained again under a lot of illumination conditions, and the 
calculations have to be done all over again. Therefore, in the 
registration step, a huge amount of images of the registered object in 
various feasible poses have to be obtained under a lot of illumination 

15 conditions, taking much time and manpower. If the target object is 
photographed in the comparison step in a different pose, the comparison 
becomes impossible. 

Moreover, the technique of the method (8) requires large amounts of 
20 calculation for obtaining the extreme rays, depending on the complexity 
of the shape. According to the Document 7, when there are m 
independent normal vectors on the surface of the object, the number of 
the extreme rays amounts to m(m-l) at the maximum. Therefore, an 
enormous amount of extreme rays have to be calculated unless the shape 
25 of the object is very simple as a block. Therefore, it is very difficult to 
calculate all the extreme rays for ordinary objects having complex shapes. 
Further, the method can not be directly applied to cases where the object 
has a concavity, and thereby, a cast shadow occurs. 

30 In the technique of the method (9), the amount of calculations in the 
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nonnegative least-squares method also tends to be enormous depending 
on the number of extreme rays. In addition to the above problems, the 
sampling method also has a problem that it is not clear how many basis 
images are necessary for obtaining satisfactory result and performance. 

5 

Further, both the methods (8) and (9) have been constructed 
assuming that the surface reflectance property of each object is perfect 
scattering. Therefore, the methods can not be applied directly to cases 
where mirror reflection exists or cases where the surface reflectance 
10 property is diffusion reflection but not perfect scattering. The majority 
of ordinary objects do not exhibit the surface perfect scattering. 

Object of the Invention 

The objects of the present invention is to provide a method, device 
15 and a record medium for image comparison, wherein^ 

first, as an input image used for comparison, a two-dimensional 
image obtained by an ordinary camera can be employed; 

second, the three-dimensional changes of the position/pose of an 
object in the input image can be corrected; 
20 third, necessary data can be easily measured in the registration 

step; and 

fourth, the correction of illumination conditions can be conducted 
with regard to the input image which is obtained under various 
illumination conditions at high processing speed. 

25 

Means of Solving the Problems 

In accordance with a first aspect of the present invention, in a 
registration step for registering data of each person to be compared in a 
database, three-dimensional shape of the object and surface reflectance 
30 of the object are measured and stored. In a comparison step for 
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comparing an input image with the registered data, a two-dimensional 
image is photographed as an input image by use of video camera, etc. 
Thereby, illumination variation images being images of an object in 
predetermined certain position/pose under various illumination 
5 conditions by use of the registered three-dimensional shape and 
reflectance data, an image being most similar to the input image from 
the illumination variation images is obtained to be generated as a 
reference image, and an evaluation value concerning similarity between 
the reference image and the input image is calculated, and such 
10 processes are performed as a verification of whether or not the reference 
image is the target object, a search for a registered object corresponds to 
the target object, and a search for a registered object which is similar to 
the target object. 

15 In accordance with a second aspect of the present invention, in the 
first aspect, in the reference image generation step, an illumination 
variation space being an image space which is spanned by the 
illumination variation images is generated, and an image which is most 
similar to the input image in the illumination variation space is 

20 generated as a reference image. 

In accordance with a third aspect of the present invention, in the 
second aspect, in the illumination variation space generation step, an 
illumination variation space is generated by obtaining the basis vectors 

25 of a space that almost accommodates image variation due to the 
illumination variation by conducting a statistic analysis such as 
principal component analysis to the illumination variation images, and 
in the reference image generation step, correlations are obtained 
between the basis vectors and the input image , and the reference image 

30 that is the most similar to the input image is generated by use of the 
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basis vectors and based on the correlations. 

In accordance with a fourth aspect of the present invention, in the 
first aspect, in the registration step, the illumination variation images 
5 are preliminarily generated, and the illumination variation images are 
stored, and in the comparison step, the registered illumination variation 
images are used for generating the reference image. 

In accordance with a fifth aspect of the present invention, in the 
10 second aspect, generation of illumination variation images and an 
illumination variation space in the comparison step is preliminarily 
performed in the registration step, and the generated illumination 
variation space is stored, and the registered illumination variation space 
is used for generating the reference image in the comparison step. 

15 

In accordance with a sixth aspect of the present invention, in the 
third aspect, generation of illumination variation images and basis 
vectors in the comparison step is preliminarily performed in the 
registration step, and the generated basis vectors are stored, and the 
20 registered basis vectors are used for generating the reference image in 
the comparison step. 

In accordance with a seventh aspect of the present invention, in one 
of the first to third aspects, position/pose parameters of the object of the 
25 input image are separately inputted instead of employing the 
predetermined certain position/pose. 

In accordance with a eighth aspect of the present invention, in one 
of the first to third aspects, the position/pose of the object of the input 
30 image is estimated by use of the registered three-dimensional shape and 
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reflectance instead of employing the predetermined certain position/pose. 

In accordance with a ninth aspect of the present invention, in one of 
the first to eighth aspects, instead of measuring the reflectance in the 
5 registration step, image information is photographed under an adequate 
illumination to be used as a substitute for the reflectance in the following 
processes. 

That is, in the ninth aspect of the present invention, an image 
10 photographed under an adequate illumination condition can be 
substituted for the surface reflectance of the object. For example, there 
is employed the fact that an image being photographed under such 
illumination conditions where the object is illuminated uniformly and 
any shadow is hardly cast, its intensity value is considered to be 
15 substantially in proportion to its reflectance. 

As an example of a method capable of being implemented to an 
image photographing means substituted for the reflectance measurement 
means, a following method is present. A hemispherical scaffold is set up 
20 in front of the object to be registered and an adequate number of 
illumination lamps are fixed to the scaffold. An image is photographed 
with all lamps being turned on at the same time. Other than this 
method, there are various methods as to employing one or more 
reflectors in order to diffuse light and illuminate the object uniformly. 

25 

In accordance with the tenth aspect of the present invention, the 
reflectance measurement in the registration step in one of the first to 
eighth aspects is replaced by photographing texture images under an 
adequate number of illumination conditions being sev; so as to be 
30 equivalent to illumination conditions set in an illumination condition 
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variation process. Further, instead of varying the illumination 
conditions in the generation step of the illumination variation images in 
one of the first to eighth aspects, employing each image of the texture 
images to generate an illumination variation image. Moreover, other 
5 than the above changed points, processes are same as those of the first to 
eighth aspects. 

That is, in accordance with the tenth aspect, there is set an adequate 
number of illumination conditions for generating enough .sample images 

10 for generating illumination variation spaces including the variation of 
the image of the comparison target object due to tha illumination 
conditions. By photographing image information under the illumination 
conditions, it is possible to generate sample images for generating 
illumination variation space without the need of the reflectance 

15 measurement, illumination condition setting in the CG image generation, 
shadow generation by means of ray tracing, etc. 

As an example of a method which can be applied to a photographing 
process of texture images, there is the following method. A 

20 hemispherical scaffold is set up in front of the object to be registered and 
an adequate number of illumination lamps are fixed to the scaffold at 
even intervals. The object is photographed repeatedly by successively 
turning each lamp on. It is also possible to photograph the object 
repeatedly moving a lamp by a manipulator and turning the lamp on 

25 repeatedly. 

The photographed texture images are registered and stored together 
with the three-dimensional shape data as the registered data. The 
comparison step is conducted without executing the illumination 
30 condition variation process. Instead of the illumination condition 
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variation process, the stored texture images are successively read out 
and an image of the registered object in the position/pose estimated by 
the position/pose estimation process is generated by means of computer 
graphics so that intensity values on the surface of the object in the 
5 generated image will be the same as those of the texture images. The 
generated image is outputted as the sample image. The following 
processes are same as those of one of the first to eighth aspects of the 
present invention. 

10 In accordance with a eleventh aspect of the present invention, in 

one of the first to tenth aspects, in the registration step, instead of 
measuring three-dimensional shapes of plurality of comparison target 
objects, by measuring one or a few number of three-dimensional shapes, 
one three-dimensional shape being the one three-dimensional shape or 

15 the average of a few number of the three-dimensional shapes is 
outputted, and thereby, shapes of all target objects are not measured, 
and measurement of the reflectance is performed in such a way that the 
reflectance of all comparison target objects is measured or image 
information is photographed under one or a plurality of illumination 

20 conditions, and only the average shape data as three-dimensional data in 
the following processes. 

That is, in accordance with the eleventh aspect of the present 
invention, the position/pose estimation process and the illumination 
25 correction process can be performed by use of the representative 
three-dimensional shape data instead of measuring and using the 
three-dimensional shapes of all objects in the case where the shapes of 
the objects are similar to each other. 

30 The following method is used as an example of a method which is 
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adaptable to the generation of an average three-dimensional shape. 
Here, a description will be given of an example for obtaining an average 
shape of two objects. The three-dimensional shapes of an object 1 and 
an object 2. The three-dimensional shape data of the registered object 1 
5 or 2 is translated so that the barycenters of the objects 1 and 2 will 
overlap one another as shown in a left diagram in Fig. 9, and sections 
that are perpendicular to the z-Rxis at appropriate intervals are set, and 
average shapes on each of the sections are calculated. As shown in a 
right diagram in Fig. 9, a line being average calculation line is drawn 
10 outward from the barycenter of the object on the section, and intersection 
points Pi and P2 of the average calculation line with shapes of the objects 
1 and 2 are obtained. The three-dimensional coordinates of a point P m 
of the average shape are obtained by averaging three-dimensional 
coordinates (xj, yi, zj) and (aj?, y2, zb> of the surface points Pi and P2 as 

15 expressed by — — — ~ - By performing the above process by 

^ 2 2 2 ) 

repeating the above calculation around the barycenter at proper 
intervals, the average shape of the objects 1 and 2 can be generated. 
Thereby, in the three-dimensional shape measurement; process, the 
generated average shape can be outputted. 

20 

This method can be adopted to the case where the number of the 
objects is more than three. Further, various other method can also be 
employed to obtain the average shape. 

25 In the following, in a process in a comparison step, the average 

shape is read out from the data storage process and substitutes for the 
three-dimensional shape of each object. 



In accordance with a twelfth aspect of the present invention, in one 
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of the first to eleventh aspects, the image comparison method does not 
include the three-dimensional measurement process and the 
three-dimensional shape is separately inputted from drawings and the 
like. 

5 

That is, in the twelfth aspect of the present invention, as an example, 
in cases where the three-dimensional shape measurement by use of an 
ordinary three-dimensional shape measurement device is difficult and 
actually making a survey or measurement is necessary, for example the 
10 case where the comparison target object is an industrial product with a 
design drawing, a house, a building, etc., it is possible to input the 
three-dimensional shape of an object from drawings and the like. 

In accordance with a thirteenth aspect of the present invention, in 
15 the image comparison method in one of the first to eleventh aspects, the 
reflectance is not measured and reflectance is inputted separately from 
drawings and the like. 

That is, in the thirteenth aspect of the present invention, as an 
20 example, in the case where the comparison target object whose surface 
reflectance of each position on the painted surface of the object such as 
industrial products is known, the three-dimensional shape of the object 
can be inputted from drawings and the like. 

25 In accordance with the fourteenth aspect of the present invention, 

in the image comparison method described in one of the first to 
thirteenth aspects, as information necessary for performing a 
position/pose estimation process, position information of a feature point 
of the comparison target object is added to the registered data, in the 

30 registration step. 
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That is, in accordance with the fourteenth aspect of the present 
invention, there is employed the fact that positions of the feature points 
of the comparison target object in the input image are extracted, and the 
5 three-dimensional coordinates of the feature points can be obtained from 
the registered data, and thereby, position/pose parameters of the object 
in the input image and photographing device parameters can 
automatically be obtained by use of the positions of the feature points on 
the image and the three-dimensional coordinates. 

10 

As a method for automatically obtaining the parameters concerning 
the position/pose and the photographing device when positions of points 
in an image and three-dimensional coordinates of the points are known, 
a camera calibration method can be used. There are various methods of 
15 the camera calibration method, and as an example thereof, the method in 
Document 15- Roger Y. Tsai "An Efficient and Accurate Camera 
Calibration Technique for 3D Machine Vision", Proc. CVPR '86, 
pp364-374 (1986), can be employed. A detailed description thereof is 
omitted here. 

20 

Embodiments of the Invention 

The present invention employs the following principles that an 
image of an object in an arbitrary position/pose under an arbitrary 
illumination condition can be generated by preliminarily registering the 
25 three-dimensional shape and surface reflectance of the object, and a set 
of images of the object in the position/pose under various illumination 
conditions can be expressed as a low-dimensional subspace of the image 
space. 

30 A description will be given of a position/pose estimation process. In 
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the position/pose estimation process, there is estimated photographing 
conditions such as parameters of position/pose of the object and 
photographing device at the time when the input image was 
photographed. As the position/pose estimation process, various methods 
5 which are performed automatically or manually can be employed. 

For example, such method can be employed as manually performed 
method that when the input image and an image generated by using 
computer graphics (hereinafter referred to as CD image) from the 
10 three-dimensional shape of the comparison target object and reflectance 
thereof are overlapped, both images are overlapped as exactly as possible 
by employing an interactive interface by which position/pose parameters 
can be adjusted or inputted. 

15 As a method for performing estimation of position/pose, there can be 

employed the method in which CG images of the comparison target 
object in various positions/poses are successively generated, and the CG 
images are compared with the input image, and a CG image that 
matches the input image the best is selected and thereby the parameters 

20 of position/pose and the photographing device can be obtained. 

Further, instead of comparing images, the position of the points or 
areas having features (hereinafter referred to as feature points) where 
intensity changes steeply etc. on an image is detected in the input image 
25 and CG images, and thereby, parameters concerning the position/pose 
and the photographing device are calculated by selecting a CG image 
whose feature point positions are the nearest to those of the input image. 

It is also possible to determine the position/pose of the target object 
30 by use of the relationship between the feature point positions in the 
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input image. In Document 13: Shinn-Ying Ho and Hui-Ling Huang "An 
analytic solution for the pose determination of human faces from a 
monocular image", Pattern Recognition Letters, Vol.19, pp. 1045- 1054 
(1998), feature points such as the tails of eyes and the ends of lips are 
5 used when human faces are used as a comparison target object, and the 
position/pose of an object is determined by use of the relationship 
between the feature points such that a line connecting two eye-feature 
points is parallel to a line connecting two mouth-feature points, etc. 

10 As the fourteenth aspect of the present invention, by registering the 

positions of the feature points of the object, the method of the camera 
calibration can be used. There are various methods of the camera 
calibration method, and as an example thereof, the method in Document 
15: Roger Y. Tsai "An Efficient and Accurate Camera Calibration 

15 Technique for 3D Machine Vision", Proc. CVPR '86, pp364-374 (1986), 
can be employed. 

Next, a description will be given of an illumination correction 
process. 

20 

Assuming that the reflectance property on the surface of the object is 
perfect scattering and the surface is convex with no shadow caused by 
light occlusion by other parts and the light source exists at an 
infinite -point, the intensity value Ku, v) of each pixel (u, f) of an image 
25 can be expressed as the following [Equation l], by use of the reflectance 

iKu, v) and a normal vector N (u, v) corresponding to the pixel (w, v) 

— > 

and the intensity Ii and a direction vector L. : 
[Equation l] 
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( ( -> -* 
l(u 9 v)=B{u 9 v)max /, A^/,v) 



Here, if we neglect the effect of the "max 0\ an arbitrary 

illumination condition including cases where a plurality of lamps are 

— > 

5 used can be expressed by only one illumination vector Las follows' 
[Equation 2] 

/( W ,v) = 5(//,v)iV( W ,v)-Z^ = Li r j 

The degree of freedom of images of the object generated by the 
10 illumination variation is equal to the number of dimension of the 

illumination vector L, that is, 3 or less. However, the degree of freedom 
actually becomes higher due to the effects of the "max 0\ cast shadow 

occurring on the surface of the object due to light occlusion by other parts, 

imperfect scattering on the surface, etc. However, most of the images 

15 lie in or he sufficiently close to a three-dimensional subspace. Therefore, 

actually occurring image variation can be approximated as a 

low-dimensional subspace. The low-dimensional subspace will be 

hereinafter referred to as an "illumination variation space 11 of each 

object. 

20 

The principal component analysis is used for obtaining basis vectors 
of the illumination variation space. A plurality of images of the target 
object generated by the illumination variation (hereinafter, referred to as 
"illumination variation images") are prepared, and a set being composed 
25 of all the images generated by the variation of illumination condition is 
approximated. Each of the illumination variation images is generated 
as an image under illumination of a single point source of light at an 
infinite-point. A plurality of illumination variation images are prepared 
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by setting and changing the direction of the point source of light at 
appropriate intervals so that all possible directions of the point sources 
in the photographing of the input image can be covered. An image 
obtained under plurality of illumination lamps can be expressed as the 
5 sum of images obtained under each single illumination lamp, therefore, 
the illumination variation images each of which is generated under a 
single point source of light are enough. For the generation of the 
illumination variation images, the three-dimensional shape data and the 
surface reflectance data which have been stored are used. The 

10 illumination variation images can be generated by use of basic functions 
of computer graphics, for example. The details of the computer graphics 
functions have been described in a document- Mason Woo, Jackie Neider, 
and Tom Davis "Open GL programming Guide", Addison-Wesley 
Publishers Japan. In a standard function provided to computers, the 

15 reflectance property of the surface of the object is generally assumed to 
be perfect scattering and only shade is generated. However, in the 
present invention, a reflectance model that is close to actual surface 
reflectance property of the object is selected and used. The mirror 
reflection is taken into consideration and cast shadows are also 

20 reproduced by means of ray tracing, in order to generate images that are 
close to reality. 

The above illumination variation image generation method by use of 
the CG functions of a computer is only an example. It is of course 
25 possible to generate each illumination variation image by calculating 
intensity of pixels that will be used for the comparison, by means of 
numerical calculations. 

Hereafter, each image will be expressed by use of a column vector. 
30 Each component of the column vector indicates an intensity value of each 



pixel in an area of the image that will be used for the comparison. 
When the number of the illumination variation images is N, by 

— > 

expressing each illumination variation image by a vector AT. U = 1, 

2, • ■ • , A), a covariance matrix Vcan be expressed by the following 
5 [Equation 3]. 
[Equation 3] 



N 



V = — SS T 
N 



10 Eigenvalues oi and eigenvectors B i U = 1, 2, • * • M) of Fare 

successively obtained for M largest eigenvalues starting from the largest 
eigenvalue, and the illumination variation space of an object "f is 
approximated by an Mr dimensional linear space Wj whose bases are the 

eigenvectors { B i }. The dimension number M of the illumination 

15 variation space may be determined in consideration of precision that is 
required of the illumination correction process. In the case where M 
eigenvectors are used, "cumulative contribution" of the M eigenvalues 
can be calculated by [Equation 4] as follows • 
[Equation 4] 

20 ^~ ]<Ji xl00[%] 

The cumulative contribution is an index indicating how precisely the 
illumination variation space can express the illumination variation 
images, when error between images is evaluated by use of the differences 
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of intensity values therebetween. By setting a threshold value for this 
index, the dimension number M can be determined automatically by 
finding a number M necessary for surpassing the threshold value. 

5 Comparison processing of an object may be performed j'by use of the 

distance of illumination variation space between an input image vector 

T q and a comparison target object as a measure. This distance exists in 

the illumination variation space Wj\ and is calculable as the distance 

— > 

between the input image and an image vector I c being closest to the 

10 input image. While various measures can be used as a measure of a 
distance, a description will be given of an example in which a square 
error of intensity value is directly used, here. 

— > 

In the case where this distant measure is used, the image vector I c 

15 being closest to the vector I q in the space Wj can be generated by the 

following [Equation 5]. 
[Equation 5] 

I c =Y^I q -BijBi 

A distance Z>(a sum of square of intensity value) between a reference 

— > — v 

20 image vector I c and the input image vector I q can be calculated by the 

following [Equation 6]. 
[Equation 6] 



D = 



Ia-Ic 



2 
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The value D is an evaluation value of similarity between the input 
image and the registered data, and based on this evaluation value, 
judgment processes such as whether or not the target image is the 
5 registered image, to which registered images the input image 
corresponds, to which registered image the input image is similar, are 
performed. For example, for simply judging whether or not the target 
image is the registered image, a threshold value D' is predetermined, 
and the target object is judged to be the registered object if D < D\ 

10 

Practical Example of the Invention 

Fig. 1 is a block diagram showing the composition and procedure of 
an image comparison device in accordance with a first embodiment of the 
15 present invention. Fig. 2 is a composition diagram of this embodiment. 
This embodiment corresponds to the invention of claim 25. 

As shown in Fig. 1, a registration means measures the 
three-dimensional shape of each object and the reflectance or color on the 
20 surface of the object by use of a three-dimensional shape measurement 
device, and stores both of the data. A comparison means photographs a 
two-dimensional image by use of a video camera, etc., and downloads the 
image into a processor which performs a comparison process, and 
performs a comparison process. 

25 

First, a registration means 100 registers three-dimensional shape 
and surface reflectance of an object as registered data to be used for 
comparison of objects. 

30 A three-dimensional shape measurement means 110 measures the 
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three-dimensional shape of each object. For example, a 
three-dimensional shape measurement device which has been disclosed 
in Japanese Patent Application No.HEI 11- 123687 can be employed. 
Other three-dimensional shape measurement devices can be employed. 

5 

A reflectance measurement means 120 measures the reflectance of 
the surface of the object corresponding to the three-dimensional shape. 
For, example, if the three-dimensional shape measurement device 
disclosed in Japanese Patent Application No.HEI 11- 123687 is employed, 
10 color information on the surface of the object can be measured 
simultaneously with the three-dimensional shape. Hereinafter, the 
color information is substituted for the reflectance data. 

A data storage means 130 stores the three-dimensional shape data 
15 and the reflectance data of each registered object. The data stored in 
the data storage means 130 are read out at a proper timing for the 
comparison process by a comparison means 200. 

An object comparison process with images is performed in the 
20 comparison means 200 with respect to the object processed by the 
registration means 100. 

In the comparison means 200, an input image beinjf a comparison 
target is photographed by use of a photographing method such as a 
25 camera, and the obtained image is captured by a processor being a 
comparison means. 

A photographing means 210 photographs the input image being the 
comparison target by means of a photographing means such as camera 
30 and video camera. 
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In a position/pose estimation means 220, parameters concerning 
position/pose or a photographing device being photographing condition 
when the input image is obtained. For example, the position/pose 
5 parameters include the translation distance (Tx, Ty t Tz) of the object, 
rotation angles (Ex, By, Rz), the focal length of the camera, and the 
viewing angle " a". The position/pose estimation means 220 is provided 
with an interactive interface by which the user adjusts the position/pose 
parameters manually watching a display. For example, an image of a 

10 comparison target object generated by computer graphics by use of the 
above 8 position/pose parameters and the input image are displayed by 
being superimposed on the screen. The user successively adjusts the 
parameters so that the two images will be in register and thereby 
appropriate parameters are determined. The interactive interface is 

15 only an example and various types of devices can be employed. It is also 
possible that position/pose parameters are automatically calculated. 

The illumination correction means 230 generates, by use of the 
above parameters determined by the position/pose estimation means 220, 
20 an image, as a reference image, which is in the same position/pose as 
that of the input image and is under an illumination condition that is the 
most similar to that of the input image. Fig. 3 is a block diagram 
showing the detailed composition and process flow of the illumination 
correction means 230. 

25 

An illumination condition variation means 231 sets a sufficient 
number of illumination conditions capable of approximating the 
illumination variation space of the object. For example, considering one 
light source at an infinite-point, and the direction of a point source of 
30 light is expressed by angles ( 6 , <z0 which indicate a longitude 9 and a 



37 



latitude ^ of a spherical surface of an object such as shown in Fig. 4. 
Each angle ( &\ <f>) is changed from -90° to 90° at intervals of 10° , 
and thereby, 361 types of illumination conditions are set. The type of 
light source, the interval or the range of illumination directions is only 
an example, and other settings can of course be employed. 

An image generation means 232 reads the three-dimensional shape 
data and the reflectance data of the object j being a comparison target 
from the data storage means 130, and generates the illumination 
variation images by use of the position/pose parameters supplied from 
the position/pose estimation means 220 and the illumination conditions 
supplied from the illumination condition variation means 231 by use of 
CG functions. The above process can be implemented by use of basic 
functions of a computer being provided with graphics functions, for 
example. In the image generation by use of computer graphics, various 
types of surface reflectance models and camera models can be employed. 
For example, the pinhole camera model may be employed as the camera 
model and the perfect scattering model can be employed as the surface 
reflectance model. It is also possible to give shadows to the surface of 
the object by means of ray tracing, or to give specularity to the surface of 
the object by use of other reflectance models. In the image comparison 
process, reflectance property model, a light source model and a camera 
model are made to be close to reality, comparison quality can be 
improved. The image generation process can also be conducted by 
numerical calculations without using computer graphics. 

The illumination variation space generation means 233 calculates 
illumination variation space from the illumination variation images by 
following the [Equation 3], and outputs the obtained basis vectors as the 
illumination variation space (basis vectors) WJ. In this embodiment, M 
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basis vectors corresponding to the M largest eigenvalues are obtained 
starting from the largest eigenvalue, and the obtained M basis vectors 
are outputted as the illumination variation space WJ. In order to 
determine the number of the basis vectors M 7 for example, as the number 
5 which surpass 95% of the cumulative contribution being calculated by 
the [Equation 4], in the case where the number of pixels equals to or less 
than 361 being the number of the illumination variation images, the 
number of pixels is iVand eigenvalues are obtained, and a number M 

that satisfies the ^ 1 >0.95 is obtained. M can be determined by 

Em* 7 ' 

10 applying various criterions. 

An illumination condition estimation means 234 generates, as a 
reference image, which is the nearest to the input image and is in the 
illumination variation space ^according to the [Equation 5]. 

15 

An image comparison means 240 calculates an evaluation value 
concerning the similarity between the input image and the generated 
reference image. Various calculation methods can be employed for the 
evaluation. For example, the sum of squares of difference of intensity 

20 value of each pixel of an image which is shown in the following [Equation 
6] can be used as the evaluation value. A technique elaborated on in a 
document- Shigeru Akamatsu "Computer Recognition of Human Faces - 
A Survey - M , The Transactions of the Institute of Electronics, Information 
and Communication Engineers D- II , Vol. J-80-D- n , No.8, pp.2131-2146 

25 (1997), can also employed. 

A comparison judgment means 250 judges whether or not the 
comparison target image is the registered image by processing the 
calculated evaluation value as a threshold value. When a plurality of 
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objects have been registered, the process from 230 to 240 are repeated for 
a plural times. In this case, the judgment means 250 can judge which 
registered object is the most similar to the target object. It is also 
possible to search for one or more registered objects having more than 
5 certain evaluation values as objects similar to the target object. 

Next, a detailed description will be given of a second embodiment of 
the present invention with reference to Figs. 6 and 7. This embodiment 
corresponds to the invention claimed in claim 26. 

10 

Fig. 6 is a block diagram showing the composition and operation of 
an image comparison device in accordance with the second embodiment 
of the present invention. This embodiment differs from the first 
embodiment in that by photographing images under a plurality of 
15 illumination conditions, and by use of the images instead of the 
reflectance, illumination variations images are generated, instead of 
measuring the reflectance by use of the reflectance measurement means 
120 in the first embodiment. Further, in the second embodiment, the 
illumination condition variation means 231 is not provided. 

20 

In a registration means 2100, as registration data to be used for 
comparison of the object, the three-dimensional shape of each object and 
image data of each object obtained under a plurality of illumination 
conditions are registered. 

25 

The three-dimensional shape measurement means 2110 measures 
the three-dimensional shape of each object by use of a three-dimensional 
shape measurement device such as the device disclosed in Japanese 
Patent Application No.HEIll-123687. 
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The texture image photographing means 2120 actually sets 
illumination conditions which are equivalent to those set and outputted 
by the illumination condition variation means of the first embodiment, 
and photographs each object under the illumination conditions, and 
5 outputs them as texture images. For example, a hemispherical scaffold 
is set up in front of the registered object and an appropriate number of 
illumination lamps are fixed to the scaffold. Concretely, in the angles 
( 0 , 0) shown in Fig. 4 with respect to the object, the lamps are fixed to 
points corresponding to angles from -90° to 90° at intervals of 15° 
10 for example, and the object is photographed repeatedly by successively 
turning each lamp on. It is also possible to photograph the registered 
object repeatedly moving a lamp by a manipulator and turning the lamp 
on repeatedly. The images obtained as above are outputted as texture 
images. 

15 

A data storage means 2130 stores the three-dimensional shape data 
of the objects and the texture images. The registered data are read out 
at the time of the comparison process by the comparison means. 

20 In a following comparison means 2200, an object comparison process 

with images are performed with respect to the object processed by the 
above comparison means 2100. 

A photographing means 2210 and a position/pose estimation means 
25 2220 are each performs the same processes as the photographing means 
210 and the position/pose estimation means 220 in the first embodiment. 

The illumination correction means 2200 does not include the 
illumination condition variation means 231, and therefore, in an image 
30 generation means 2232, illumination variation images are generated by 
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use of, as intensity value of a surface of the object, the texture images 
being photographed by the texture image photographing means 2120. 
Fig. 7 is a block diagram showing the process flow of an illumination 
correction means according to the second embodiment. 

5 

The image generation means 2232 reads out the three-dimensional 
shape data and the texture images of a comparison target object n f from 
the data storage means 2130, and generates illumination variation 
images by computer graphics, by use of the position/pose obtained by the 

10 position/pose estimation means 2220 and each of the texture images. 
The process can be conducted by means of texture mapping, which is a 
basic function of a computer having graphic functions. In the second 
embodiment, various types of camera models such as the pinhole camera 
model can be employed. The texture images employed in the second 

15 embodiment are actually photographed images, and therefore, there is no 
need to generate shadows and specularity by computer graphics as in the 
first embodiment. 

An illumination variation space generation means 2233 and below 
20 performs the same processes as those of the first embodiment. 

In the following, a detailed description will be given of a third 
embodiment of the present invention. This embodiment corresponds to 
the invention claimed in claim 27. 

25 

Fig. 8 is a block diagram in accordance with the third embodiment of 
the present invention. In this embodiment, instead of the 
three-dimensional shape measurement means 3110 which measures the 
three-dimensional shapes of all objects when registering a plurality of 
30 objects in the first embodiment, by measuring the three-dimensional 
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shapes of one or a small number of objects, an average shape generation 
means 3150 generates an average three-dimensional shape of the 
measured objects. The third embodiment is different from the first 
embodiment in that shapes of all the comparison target objects are not 
5 measured, and that a comparison means 3200 uses the 
three-dimensional shape of the average shape. 

First, in the three-dimensional shape measurement means 3110, as 
registered data used for comparison of objects, three-dimensional shape 
10 and image data under various illumination conditions with respect to 
objects 1 and 2 are registered. 

The three-dimensional shape measurement means 3110 measures 
three-dimensional shapes of the objects 1 and 2, by use of such a 
15 three-dimensional shape measurement device as disclosed in Japanese 
Patent Application No.HEI 11* 123687. 

The average shape generation means 3150 translates the 
three-dimensional shape data of the registered object 1 or 2 so that the 

20 barycenters of the objects 1 and 2 will overlap one another as shown in a 
left drawing of Fig. 9, sets sections that are perpendicular to the ^axis at 
appropriate intervals, and calculates average shapes on each of the 
sections. As shown in a right drawing in Fig. 9, a line being average 
calculation line is drawn outward from the barycenter of the object on 

25 the section, and intersection points Pi and P2 of the average calculation 
line with shapes of the objects 1 and 2 are obtained. The 
three-dimensional coordinates of a point P m of the average shape are 
obtained by averaging three-dimensional coordinates (xj, yi, zj) and (xz, 
ys, zz) of the surface points Pi and P2 as expressed by 
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By performing the above process by repeating the 



2 2 2 

above calculation around the barycenter at proper intervals, the average 
shape of the objects 1 and 2 can be generated. Thereby, in the average 
shape measurement means 3150, the generated average shape can be 
outputted. 



The reflectance measurement means 3120 measures the reflectance 
of the surface of each object 1 and 2 corresponding to the 
three-dimensional shape. If the three-dimensional shape measurement 

10 device of Japanese Patent Application No.HEI 11- 123687 is employed, 
color information on the surface of the object can be measured 
simultaneously with the three-dimensional shape. In the following, the 
color information is substituted for the reflectance data. The 
correspondence between each point of the average throe-dimensional 

15 shape and the color information of each registered object can be set as 
follows. That is, if we express the three-dimensional shape data of the 
objects 1 and 2 which have been used for the calculation of the 
three-dimensional coordinates of the average three-dimensional shape 
P m as Pi and P2, reflectance data of the object 1 corresponding to the P m 

20 is the reflectance corresponding to the Pi, and in the same way, 
reflectance data of the object 2 corresponding to the average shape P m is 
the reflectance corresponding to P2. 

The data storage means 3130 stores the average shape inputted 
25 from the three-dimensional shape measurement means 3110 and the 
reflectance of the objects 1 and 2 inputted from the reflectance 
measurement means 3120. 



The processing of the comparison means 3200 is basically the same 
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as that of the first embodiment, except that the average 
three-dimensional shape data read out from the data storage means 3130 
is changed to the average shape of each of objects 1 and 2. 

5 While an embodiment in which an average shape of two objects is 

stored when two objects are registered has been explained above, this 
embodiment is only for an example. An embodiment in which the 
number of objects is more than three and arbitrary number thereof are 
calculated for obtaining the average shape is also possible by the same 
10 processing. 

In the following, a detailed description will be given of a fourth 
embodiment of the present invention. This embodiment corresponds to 
the invention claimed in claim 30. In the fourth embodiment, human 
15 face is used as an example of a comparison target object. 

Fig. 10 is a block diagram showing the composition and process flow 
of an image comparison device in accordance with this embodiment. 
Differently from the image comparison device of the first embodiment, a 

20 registration means 4100 is further provided with a feature point 
extraction means 4140 which extracts feature points where intensity etc. 
changes steeply from each object and outputs the extracted points as the 
feature point positions. A data storage means 4130 of the fourth 
embodiment also stores the feature point positions. In a comparison 

25 means 4200 of the fourth embodiment, a position/pose estimation means 
4220 reads out the feature point positions from the data storage means 
4130 and automatically executes the position/pose estimation. 

First, in the registration means 4100, the three-dimensional shape 
30 and reflectance of each object are measured as the registered data to be 
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used for the object comparison, and three-dimensional coordinates of 
feature points of the object are obtained by use of the measured 
three-dimensional shape and reflectance. The three-dimensional shape, 
the reflectance or color information and the feature point positions are 
5 registered. 

The three-dimensional shape measurement means 4110, which is 
provided with a three-dimensional shape measurement device, measures 
the three-dimensional shape of each object. While the 

10 three-dimensional shape measurement device of Japanese Patent 
Application No.HEI 11- 123687 is employed in the fourth embodiment as 
an example, other devices can of course be used. 

The reflectance measurement means 4120 measures the reflectance 
15 of the surface of the object corresponding to the three-dimensional shape. 
If the three-dimensional shape measurement device of Japanese Patent 
Application No. HEIll- 123687 is employed, color information on the 
surface of the object can be measured simultaneously with the 
three-dimensional shape. The color information obtained by the device 
20 will hereafter be substituted for the reflectance. 

The feature point extraction means 4140 detects points or areas 
where intensity etc. changes steeply (hereinafter referred to as feature 
points) of the object on an image, and outputs the three-dimensional 

25 coordinates of the feature points (feature point positions) in the data 
storage means 4130. For example, when human faces are comparison 
target objects, parts where reflectance changes steeply such as tails of 
eyes, lips, etc., parts where the three-dimensional shape changes steeply 
such as tip of nose etc., are detected. The feature point detection 

30 process can be conducted by manual operation. For the automatic 
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feature point extraction process, various methods including those 
disclosed in the aforementioned Japanese Patent No.2872776- "face 
image comparison device" and the aforementioned Japanese Patent 
Application Laid-Open No. HEI6- 168317: "personal identification device" 
5 can be employed. In this embodiment, twelve points (0 ~ 11) shown in 
Fig. 12 are used as the feature points. The definition of the feature 
points can of course be altered depending on what the comparison target 
object is. In the following explanation, the feature point positions being 

-> . 

three-dimensional coordinates will be expressed by use of vectors A i = (xj, 
10 y u zh U= 0, 1, • • • , 11). 

The data storage means 4130 stores the three-dimensional shapes, 
the reflectance and the feature point positions of the registered objects. 
The registered data is read out when the comparison step is conducted. 

15 

In the following comparison means 4200, an object comparison 
process is performed with respect to the object processed by the 
registration means 4100. 

20 The photographing means 4210, which is provided with a camera, a 

video camera, etc., photographs the comparison target object and thereby 
obtains the input image. 

The position/pose estimation means 4220 estimates the parameters 
25 concerning the position/pose of the object and parameters concerning the 
photographing device being the condition at the time when the target 
object was photographed. Fig. 11 is a block diagram showing the 
composition and process flow of the position/pose estimation means 4220. 
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The input image feature point extraction means 4221 extracts, from 
the input image, feature points being the same position as feature point 

vectors A. extracted by the feature point extraction means 4140 of the 

— > 

registration means 4100, and outputs position vectors on the image B i = 

5 (it,, vi) (i = 0, 1, * • • , 11) to the position/pose calculation means 4222 as 
input image feature point positions. The input image feature point 
extraction process can be conducted by manual operation of an operator 
seeing the input image on the screen. The methods employed by the 
feature point extraction means 4140 such as the methods disclosed in the 

10 aforementioned Japanese Patent Application Laid-Open 
No.HEI6-168317: "personal identification device" can be used. While 
human face comparison is taken as an example in this embodiment, the 
image comparison device of the fourth embodiment can be used for the 
comparison of polyhedron-like objects, for example, by using apexes of 

15 the objects as the feature points. Edges are extracted from each object 
in the image, and the apexes of the object are detected as intersection 
points of the edges. When some characteristic patterns exist on the 
surfaces of the objects, the positions of the patterns can be used. 

20 The position/pose calculation means 4222 calculates parameters 

concerning the position/pose of the target object in the input image and 
parameters concerning the photographing device by use of the feature 
point positions read out from the data storage means 4130 and the input 
image feature point positions, and outputs the calculated parameters as 

25 the position/pose. While various method such method as of Document 
15 can be employed, in this embodiment, the position/pose parameters 
employed in this embodiment include, for example, the translation 
distance (Tx, Ty y Tz) of the object, rotation angles (fix, fiy, Rz) of the 
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object around the jraxis, j^axis and ^-axis and the focal length "f of the 
camera. The pinhole camera model is employed as the camera model. 

The relationship between the feature point position vector A i and the 

— > 

input image feature point position vector B t can be expressed by the 

5 following [Equation 7]. 
[Equation 7] 



10 



15 



20 



V 


= / 


a 




c 





where "a", n b" and V are defined as following [Equation 8]- 
[Equation 8] 
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where "R % is the following rotation matrix expressed by the [Equation 9]: 
[Equation 9] 



72 = 



cosi? v cosi?, 

y 

cosR y smR 2 
- sin R„ 



- cosR x sin R. + sin R x sin R y cosR. sin R, sin R x + cos^ sin R y cosR. 
cosR x cosR : + sin R x sin R sin R z 



sin R„ cos/? M 



- sin R x cosR z + cosR x sin R y sin R, 
cosR,. cos/?„ 



The position/pose parameters Rx, Ry, Rz, Tx, Ty, Tz, f by 
optimization calculation so that the sum of the differences between the 
value obtained by the [Equation 7] and the input image feature point 
positions with respect to the 12 feature points will be the smallest. The 
optimization calculation can be executed in various ways. The obtained 
Rx, Ry, Rz, Tx, Ty, Tz, /are outputted as position/pose parameters. The 
definitions of the position/pose parameters and the camera model and 
the calculation method explained above are only examples, and thus, the 
position/pose calculation process can be conducted according to various 
methods. 
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Processes of the illumination correction means 4230 and below are 
same as those of the first embodiment. 

5 In the following, a detailed description will be given of an image 

comparison device in accordance with a fifth embodiment of the present 
invention. This embodiment corresponds to the invention claimed in 
claim 28 and 29. 

10 In this embodiment, the objects to be registered are industry 

products etc. whose design drawings of the shapes have been saved as 
CAD data and whose surface painting specifications etc. have been 
designated and written in drawings. Differently from the first 
embodiment, a three-dimensional shape measurement means 110 of the 

15 fifth embodiment obtains the three-dimensional shape data from the 
CAD data of design drawings, and a reflectance measurement means 120 
of the fifth embodiment obtains the reflectance data by reading 
drawings. 

20 The three-dimensional shape measurement means 110 reads the 

CAD data of the of the design drawings, converts the CAD data into a 
data format that can be handled by the comparison means 200, and 
outputs the converted data. The reflectance measurement means 120 
reads data concerning the colors, surface finishing conditions, etc. of 

25 parts of the objects from the design drawings, converts the data into 
reflectance data, and outputs the data. The data storage means 130 
stores the outputted three-dimensional shape data and the reflectance 
data. The comparison means 200 of the fifth embodiment conducts the 
comparison step in the same way as the first embodiment. 



30 
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As described in the above embodiment, the present invention can be 
applied to general objects. Especially, the present invention can be 
applied to the comparison of the type and form of automobiles, and also 
the comparison of human faces. 

5 

As set forth hereinabove, while a detailed description has been given 
of the embodiments of the present invention with reference to the 
drawings, it is not to be restricted by those embodiments, and change or 
modification of the embodiment is to be appreciated without departing 
10 from the subject matter of the present invention. The present invention 
can also be implemented by use of a computer program. 

Effect of the Invention 

According to the present invention, three-dimensional shapes of 

15 objects and reflectance of surface of the objects or images under 
appropriate conditions of illumination are measured only by the 
registration means, and photographing device such as video camera for 
photographing general two-dimensional images is enough as a 
photographing means in a comparison means, and thereby, a practical 

20 device can be constructed without a need of a three-dimensional shape 
measurement device. Also, three-dimensional shapes are registered, 
and therefore, perfect correction is possible with respect to the change of 
the three-dimensional position/pose in the input images. Further, 
perfect correction is possible with respect to the variation of illumination 

25 conditions. The present invention can be applied to the target objects 
whose surface reflectance properties are not perfect scattering, and also 
applied to the target objects which cast shadows due to light occlusion by 
other parts, specularity, etc. occurring on the surfaces. Thus, the 
present invention can be applied to comparison of general objects in 

30 wider range than the above methods (8) or (9). Moreover, illumination 
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variation images can automatically be generated by use of registered 
three-dimensional shapes and reflectance, and accordingly, the 
registration step can be performed easily since photographing a lot of 
images is not needed. Furthermore, by use of the cumulative 
5 contribution, an enough dimension number of subspace as an 
approximation of the illumination variation space can be obtained, and 
therefore, enough accuracy of the image comparison can be obtained. 

Brief Description of the Drawings 
10 [Fig. 1] is a block diagram showing the process flow of an image 
comparison device in accordance with a first embodiment of the present 
invention. 

[Fig. 2] is a constitutive diagram of an image comparison device in 
15 accordance with the first embodiment of the present invention. 

[Fig. 3] is a block diagram showing the process flow of an illumination 
correction process in accordance with the first embodiment. 

20 [Fig. 4] is a diagram for explaining angles indicating the direction of 
illuminations which are used for the setting of illumination conditions 
with respect to an object. 

[Fig. 5] is a diagram showing an example of an object comparison device 
25 using images. 

[Fig. 6] is a block diagram showing the process flow of an image 
comparison device in accordance with a second embodiment of the 
present invention. 

30 
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[Fig. 7] is a block diagram showing the process flow of an illumination 
correction process in accordance with the second embodiment of the 
present invention. 

5 [Fig. 8] is a block diagram showing the process flow in accordance with a 
third embodiment of the present invention. 

[Fig. 9] is diagrams for explaining a generation method of an average 
shape. 

10 

[Fig. 10] is a block diagram showing the process flow in accordance with 
a fourth embodiment of the present invention. 

[Fig. 11] is a block diagram showing the process flow of a position/pose 
15 estimation process in accordance with the fourth embodiment of the 
present invention. 

[Fig. 12] is a diagram showing an example of points employed as feature 
points of a target object. 

20 

[Fig. 13] is a diagram for explaining, as an example of a conventional 
image comparison technique, the composition of a technique in which 
only two-dimensional images are employed in both the registration step 
and the comparison step. 

25 

[Fig. 14] is a diagram for explaining, as an example of a conventional 
image comparison technique, the composition of a technique in which 
three-dimensional shapes are measured in both the registration step and 
the comparison step. 

30 
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[Fig. 15] is a diagram for explaining, as an example of a conventional 
image comparison technique, the composition of a technique in which 
two-dimensional images are employed in both the registration step and 
the comparison step, and standard three-dimension shape is employed in 
5 the position/pose correction step. 

[Fig. 16] is a diagram for explaining, as an example of a conventional 
image comparison technique, the composition of a technique in which 
images are photographed under a lot of position/pose conditions in the 
10 registration step and recognition is performed. 

[Fig. 17] is a diagram for explaining, as an example of a conventional 
image comparison technique, the composition of a technique in which 
two-dimensional images are photographed under a plurality of 
15 illumination conditions in the registration step, and illumination 
condition correction is performed. 

[Fig. 18] is a diagram for explaining, as an example of a conventional 
image comparison technique, the composition of a technique in which 
20 two-dimensional images are photographed under a plurality of 
illumination conditions in the registration step, and illumination 
condition correction is performed. 
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Title of Document 
Abstract 

Abstract 
Problem 

5 To provide an image comparison device, an image comparison 

method and a memory medium, where three-dimension shapes are not 
needed to be measured in a comparison step, and identification of 
whether or not a target object is a registered object, comparison of which 
registered object is the target object, and a search of similar objects are 
10 performed by use of images photographed with various position/pose of 
objects and under various illumination conditions. 

Means of Solving 

A registration means 100 measures the three-dimensional shape and 
15 surface reflectance of an object to be registered and stores them as 
registered data. In a comparison means 200 photographs an input 
image by a photographing means 210 such as camera. A position/pose 
estimation means 220 estimates the position/pose of the object in the 
input image by use of the registered three -dimension shape and 
20 reflectance. An illumination correction means 230 generates, as a 
reference image, an image of the object in the same position/pose as the 
object in the input image and under the same illumination condition as 
in the input image by use of the registered three-dimensional shape and 
reflectance. An image comparison means 240 calculates an evaluation 
25 value concerning the similarity between the reference image and the 
input image. A judgment means 250 performs comparison judgment 
based on the evaluation value. 

Selected Drawing 
30 [Fig. 1] 
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[Title of Document] Specification 

[Title of Invention] DEVICE, METHOD AND RECORD MEDIUM 

FOR IMAGE COMPARISON 
[Scope of Claim for Patent] 
5 [Claim l] An image comparison device comprising: 

a registration means for registering data of a registration target 
object as three-dimensional data; and 

a comparison means for obtaining, two-dimensional data of an 
object to be compared with the data registered in the registration means 
10 and comparing the obtained data with the data of the registered object in 
the registration means. 

[Claim 2] The image comparison device as claimed in claim 1, 
wherein the comparison means includes: 

a photographing means for photographing, as an input image, 
15 the target object to be compared with the data of registered objects 
registered in the registration means; 

a position/pose correction means for correcting the position/pose 
of the target object obtained by the photographing means; 

an illumination correction means for generating, as a reference 
20 image, an image in the same position/pose as the input image, which is 
photographed by the photographing means, and under an illumination 
condition most similar to that of the input image by use of the 
position/pose corrected by the position/pose correction means and the 
data registered in the registration means; 
25 an image comparison means for comparing the reference image 

generated by the illumination correction means with the input image 
obtained by the photographing means and thereby calculating an 
evaluation value concerning the similarity between the two images; and 
a comparison judgment means for judging whether or not the 
30 target object photographed by the photographing means is the registered 
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object registered in the registration means based on the evaluation value 
calculated by the image comparison means. 

[Claim 3] The image comparison device as claimed in claim 2, 
wherein the registration means includes^ 
5 a three-dimensional shape measurement means for measuring 

the three-dimensional shape of each object to be registered; 

a reflectance measurement means for measuring the surface 
reflectance at each position on the three-dimensional shape of the object 
to be registered; and * 
10 a data storage means for storing, as the registered data, the 

three-dimensional shapes measured by the three-dimensional shape 
measurement means and the reflectance measured by the reflectance 
measurement means. 

[Claim 4] The image comparison device as claimed in claim 2, 
15 wherein the registration means includes- 

a three-dimensional shape measurement means for measuring 
the three-dimensional shape of each object to be registered; 

an image information obtaining means for obtaining image 
information of the object by photographing the object to be registered; 
20 and 

a data storage means for storing, as the registered data, the 
three-dimensional shapes measured by the three-dimensional shape 
measurement means and the image information obtained by the image 
information obtaining means. 
25 [Claim 5] The image comparison device as claimed in claim 2, 

wherein the registration means includes: 

a three-dimensional shape measurement means for measuring 
the three-dimensional shape of each object to be registered; 

an average shape generation means for generating an average 
30 three-dimensional shape of the three-dimensional shapes measured by 



the three-dimensional shape measurement means when registering a 
plurality of objects; 

a reflectance measurement means for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 
5 to be registered; and 

a data storage means for storing, as the registered data, the 
three-dimensional shape measured by the three-dimensional shape 
measurement means, the average three-dimensional shape generated by 
the average shape generation means and the reflectance measured by 
10 the reflectance measurement means. 

[Claim 6] The image comparison device as claimed in one of claims 
3 to 5, wherein the illumination correction means includes: 

an image generation means for generating, as illumination 
variation images, images of each object in the same position/pose as the 
15 object photographed by the photographing means and under various 
illumination conditions by use of the position/pose corrected by the 
position/pose correction means and the registered data of objects in the 
registration means; and 

an illumination condition estimation means for generating an 
20 image that is the most similar to the input image obtained by the 
photographing means by use of the illumination variation images 
generated by the image generation means and outputting the generated 
image to the image comparison means as the reference image. 

[Claim 7] The image comparison device as claimed in claim 6, 
25 wherein^ 

the illumination correction means further includes an 
illumination variation space generation means for generating an 
illumination variation space which is spanned by the illumination 
variation images generated by the image generation means, and 
30 the illumination condition estimation means generates the 



image that is the most similar to the input image from the illumination 
variation space generated by the illumination variation space generation 
means and outputs the generated image to the image comparison means 
as the reference image. 

[Claim 8] The image comparison device as claimed in claim 7, 
wherein: 

the illumination variation space generation means generates 
basis vectors of a space that almost accommodates image variation due 
to the illumination variation by conducting principal component analysis 
to the illumination variation images generated by the image generation 
means, and 

the illumination condition estimation means obtains 
correlations between the input image obtained by the photographing 
means and each of the basis vectors generated by the illumination 
variation space generation means, generates the image that is the most 
similar to the input image by use of the basis vectors and based on the 
correlations, and outputs the generated image to the image comparison 
means as the reference image. 

[Claim 9] The image comparison device as claimed in one of claims 
6 to 8, wherein the illumination correction means further includes an 
illumination condition variation means for setting various illumination 
conditions and outputting the illumination conditions to the image 
generation means. 

[Claim 10] The image comparison device as claimed in claim 2, 
wherein: 

the registration means includes: 

a three-dimensional shape measurement means for measuring 
the three-dimensional shape of each registered object; 

a texture image photographing means for photographing the 
object to be registered under various illumination conditions and thereby 



obtaining texture images of the registered object; and 

a data storage means for storing the three-dimensional shapes 
measured by the three-dimensional shape measurement means and the 
texture images obtained by the texture image photographing means as 
the registered data, and 

the illumination correction means includes-* 

an image generation means for generating illumination 
variation images of each object in the same position/pose as the object 
photographed by the photographing means by use of the position/pose 
determined by the position/pose determination means and the 
three-dimensional shapes and the texture images of the object registered 
in the registration means" and 

an illumination condition estimation means for generating an 
image that is the most similar to the input image obtained by the 
photographing means by use of the illumination variation images 
generated by the image generation means and outputting the generated 
image to the image comparison means as the reference image. 

[Claim 111 The image comparison device as claimed in claim 10, 
wherein the illumination correction means further includes an 
illumination variation space generation means for generating an 
illumination variation space which is spanned by the illumination 
variation images generated by the image generation means, and 

the illumination condition estimation means generates the 
image that is the most similar to the input image from the illumination 
variation space generated by the illumination variation space generation 
means and outputs the generated image to the image comparison means 
as the reference image. 

[Claim 12] The image comparison device as claimed in claim 11, 
wherein: 

the illumination variation space generation means generates 



basis vectors of a space that almost accommodates image variation due 
to the illumination variation by conducting the principal component 
analysis to the illumination variation images generated by the image 
generation means, and 

the illumination condition estimation means obtains 
correlations between the input image obtained by the. photographing 
means and each of the basis vectors generated by the illumination 
variation space generation means, generates the image that is the most 
similar to the input image by use of the basis vectors and based on the 
correlations, and outputs the generated image to the image comparison 
means as the reference image. 

[Claim 13] The image comparison device as claimed in claim 2, 
wherein the position/pose correction means outputs a predetermined 
position/pose to the illumination correction means. 

[Claim 14] The image comparison device as claimed in claim 2, 
wherein the position/pose correction means receives inputs concerning 
position/pose parameters of the object in the input image obtained by the 
photographing means, and outputs the received position/pose to the 
illumination correction means. 

[Claim 15] The image comparison device as claimed in claim 2, 
wherein the position/pose correction means estimates the position/pose of 
the object in the input image obtained by the photographing means, and 
outputs the estimated position/pose to the illumination correction means. 

[Claim 16] The image comparison device as claimed in claim 15, 
wherein: 

the registration means includes^ 

a three-dimensional shape measurement means for measuring 
the three-dimensional shape of each object to be registered; 

a reflectance measurement means for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 



to be registered; 

a first feature point extraction means for extracting feature 
point positions based on the three-dimensional shape measured by the 
three-dimensional shape measurement means and the reflectance 
measured by the reflectance measurement means; and 

a data storage means for storing the three-dimensional shapes 
measured by the three-dimensional shape measurement means, the 
reflectance measured by the reflectance measurement means and the 
feature point positions obtained by the first feature point extraction 
means as the registered data, and 

the position/pose correction means includes- 

a second feature point extraction means for extracting, from the 
input image obtained by the photographing means, input image feature 
point positions being the same feature point positions as those stored in 
the storage means; and 

a position/pose calculation means for estimating the 
position/pose of the object photographed by the photographing means 
based on the three-dimensional shape and the feature point positions 
stored in the data storage means and the input image feature point 
positions extracted by the second feature point extraction means and 
outputting the estimated position/pose to the illumination correction 
means. 

[Claim 17] The image comparison device as claimed in claim 16, 
wherein the illumination correction means includes- 

an image generation means for generating images of each object 
in the same position/pose as the object photographed by the 
photographing means and under various illumination conditions as 
illumination variation images by use of the position/pose estimated by 
the position/pose correction means and the three-dimensional shape and 
the reflectance stored in the data storage means; and 
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an illumination condition estimation means for generating an 
image that is the most similar to the input image obtained by the 
photographing means by use of the illumination variation images 
generated by the image generation means and outputting the generated 
image to the image comparison means as the reference image. 

[Claim 18] The image comparison device as claimed in claim 17, 
wherein^ 

the illumination correction means further includes an 
illumination variation space generation means for generating an 
illumination variation space which is spanned by the illumination 
variation images generated by the image generation means, and 

the illumination condition estimation means generates the 
image that is the most similar to the input image from the illumination 
variation space generated by the illumination variation space generation 
means and outputs the generated image to the image comparison means 
as the reference image. 

[Claim 19] The image comparison device as claimed in claim 18, 
wherein-' 

the illumination variation space generation means generates 
basis vectors of a space that almost accommodates image variation due 
to the illumination variation by conducting principal component analysis 
to the illumination variation images generated by the image generation 
means, and 

the illumination condition estimation means obtains 
correlations between the input image obtained by the photographing 
means and each of the basis vectors generated by the illumination 
variation space generation means, generates the image that is the most 
similar to the input image photographed by the photographing means by 
use of the basis vectors and based on the correlations, and outputs the 
generated image to the image comparison means as the reference image. 
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[Claim 20] The image comparison device as claimed in one of 
claims 17 to 19, wherein the illumination correction means further 
includes an illumination condition variation means for setting various 
illumination conditions and outputting the illumination conditions to the 
5 image generation means. 

[Claim 21] The image comparison device as claimed in claim 1, 
wherein: 

the registration means includes: 

a three-dimensional shape measurement means for measuring 
10 the three-dimensional shape of each object to be registered; 

a reflectance measurement means for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered; 

an image generation means for generating images of each object 
15 to be registered under various illumination conditions as illumination 
variation images; 

an illumination variation space generation means for generating 
an illumination variation space which is spanned by the illumination 
variation images generated by the image generation means; and 
20 a data storage means for storing the illumination variation 

space generated by the illumination variation space generation means as 
the registered data, and 

the comparison means includes: 

a photographing means for photographing, as an input image, 
25 the target object to be compared with the data of registered objects 
registered in the registration means; 

an illumination condition estimation means for generating, as a 
reference image, the image that is the most similar to the input image 
from the illumination variation space stored in the data storage means! 
30 an image comparison means for comparing each reference image 
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generated by the illumination condition estimation means with the input 
image obtained by the photographing means and thereby calculating an 
evaluation value concerning the similarity between the two images; and 

a comparison judgment means for judging whether or not each 
of the registered objects registered in the registration means is the object 
photographed by the photographing means based on the evaluation value 
calculated by the image comparison means. 

[Claim 22] The image comparison device as claimed in claim 1, 
wherein: 

the registration means includes : 

a three-dimensional shape measurement means for measuring 
the three-dimensional shape of each object to be registered; 

a reflectance measurement means for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered; 

an image generation means for generating images of each object 
to be registered under various illumination conditions as illumination 
variation images; 

an illumination variation space generation means for generating 
basis vectors of a space that almost accommodates image variation due 
to the illumination variation by conducting principal component analysis 
to the illumination variation images generated by the image generation 
means; and 

a data storage means for storing, as the registered data, the 
basis vectors generated by the illumination variation space generation 
means, and 

the comparison means includes • 

a photographing means for photographing, as an input image, 
the target object to be compared with the data of registered objects 
registered in the registration means; 



11 



an illumination condition estimation means for obtaining 
correlations between the input image obtained by the photographing 
means and each of the basis vectors stored in the data storage means, 
generating, as the reference image, the image that is the most similar to 
the input image by use of the basis vectors and based on the correlations; 

an image comparison means for comparing each reference image 
generated by the illumination condition estimation means with the input 
image obtained by the photographing means and thereby calculating an 
evaluation value concerning the similarity between the two images; and 

a comparison judgment means for judging whether or not each 
of the registered objects registered in the registration means is the object 
photographed by the photographing means based on the evaluation value 
calculated by the image comparison means. 

[Claim 23] The image comparison device as claimed in one of 
claims 2 to 22, wherein the three-dimensional shape measurement 
means measures the three-dimensional shapes by reading data or 
drawings. 

[Claim 24] The image comparison device as claimed in one of 
claims 2 to 23, wherein the reflectance measurement means obtains the 
reflectance by reading data or drawings. 

[Claim 25] The image comparison device as claimed in one of 
claims 2 to 24, wherein the photographing means obtains the input 
image by scanning a film, a photograph or printed matter. 

[Claim 26] The image comparison device as claimed in one of 
claims 2 to 25, wherein the comparison judgment means judges which 
registered object matches the target object. 

[Claim 27] The image comparison device as claimed in one of 
claims 2 to 26, wherein the comparison judgment means searches for one 
or more registered objects that are similar to the target object. 

[Claim 28] The image comparison device as claimed in one of 
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claims 1 to 27, wherein the registered objects are automobiles. 

[Claim 29] The image comparison device as claimed in one of 
claims 1 to 28, wherein the objects to be registered are human faces. 
[Claim 30] An image comparison method comprising the steps of 
5 a registration step in which three-dimensional data of each 

object to be registered are registered; and 

a comparison step in which two-dimensional data of an object to 
be compared with the data registered in the registration step is obtained, 
and the obtained data is compared with the data registered in the 
10 registration step. 

[Claim 31] The image comparison method as claimed in claim 30, 
wherein the comparison step includes- 

a photographing step in which the comparison target object is 
photographed and thereby an input image is obtained; 
15 a position/pose correction step in which the position/pose of the 

target object obtained in the photographing step is corrected; 

an illumination correction step in which an image of each 
registered object in the same position/pose as the target object in the 
input image and under an illumination condition most similar to that of 
20 the input image is obtained as a reference image by use of the 
position/pose corrected in the position/pose correction step and the data 
registered in the registration step; 

an image comparison step in which each reference image 
generated in the image correction step is compared with the input image 
25 obtained in the photographing step and thereby an evaluation value 
concerning the similarity between the two images is calculated; and 

a comparison judgment step in which it is judged whether or not 
each registered object registered in the registration step is the object 
photographed in the photographing step based on the evaluation value 
30 calculated in the image comparison step. 
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[Claim 32] The image comparison method as claimed in claim 31, 
wherein the registration step includes: 

a three-dimensional shape measurement step in which the 
three-dimensional shape of each object to be registered is measured; 
5 a reflectance measurement step in which the surface reflectance 

at each position of the three-dimensional shape of the object to be 
registered is measured! and 

a data storage step in which the three-dimensional shapes 
measured in the three-dimensional shape measurement step and the 
10 reflectance measured in the reflectance measurement step are stored as 
the registered data. 

[Claim 33] The image comparison method as claimed in claim 31, 
wherein the registration step includes: 

a three-dimensional shape measurement step in which the 
15 three-dimensional shape of each object to be registered is measured; 

a image information obtaining step in which the object to be 
registered is photographed and thereby image information of the object 
to be registered is obtained; and 

a data storage step in which the three-dimensional shape 
20 measured in the three-dimensional shape measurement step and the 
image information obtained in the image obtaining step are stored as the 
registered data. 

[Claim 34] The image comparison method as claimed in claim 31, 
wherein the registration step includes ' 
25 a three-dimensional shape measurement step in which the 

three-dimensional shape of each object to be registered is measured; 

an average shape generation step in which an average 
three-dimensional shape is generated as the average of a plurality of the 
three-dimensional shapes measured in the three-dimensional shape 
30 measurement step when registering a plurality of objects; 
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a reflectance measurement step in which the surface reflectance 
at each position of the three-dimensional shape of the object to be 
registered is measured; and 

a data storage step in which the three-dimensional shapes 
5 measured in the three-dimensional shape measurement step, the 
average three-dimensional shape generated in the average shape 
generation step and the reflectance measured in the reflectance 
measurement step are stored as the registered data. 

[Claim 35] The image comparison method as claimed in one of 
10 claims 32 to 34, wherein the illumination correction step includes ' 

an image generation step in which images of each registered 
object in the same position/pose as the target object photographed by the 
photographing means and under various illumination conditions are 
generated as illumination variation images by use of the position/pose 
15 corrected in the position/pose correction step and the data registered in 
the registration step; and 

an illumination condition estimation step in which an image 
that is the most similar to the input image obtained in the 
photographing step is generated as the reference image by use of the 
20 illumination variation images generated in the image generation step. 

[Claim 36] The image comparison method as claimed in claim 35, 
wherein^ 

the illumination correction step further includes an illumination 
variation space generation step in which an illumination variation space 
25 which is spanned by the illumination variation images generated in the 
image generation step is generated, and 

in the illumination condition estimation step, the image that is 
the most similar to the input image is generated as the reference image 
from the illumination variation space generated in the illumination 
30 variation space generation step. 
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[Claim 37] The image comparison method as claimed in claim 36, 
wherein: 

in the illumination variation space generation step, basis 
vectors of a space that almost accommodates image variation due to the 
5 illumination variation are generated by conducting principal component 
analysis to the illumination variation images generated in the image 
generation step, and 

in the illumination condition estimation step, correlations are 
obtained between the input image obtained in the photographing step 
10 and each of the basis vectors generated in the illumination variation 
space generation step, and the image that is the most similar to the 
input image is generated as the reference image by use of the basis 
vectors and based on the correlations. 

[Claim 38] The image comparison method as claimed in one of 
15 claims 35 to 37, wherein the illumination correction step further includes 
an illumination condition variation step in which various illumination 
conditions are set and generated to be used for generating illumination 
variation images before the illumination variation images are generated 
in the image generation step. 
20 [Claim 39] The image comparison method as claimed in claim 31, 

wherein: 

the registration step includes: 

a three-dimensional shape measurement step in which the 
three-dimensional shape of each object to be registered is measured; 
25 a texture image photographing step in which the object to be 

registered is photographed under various illumination conditions and 
thereby texture images are obtained; and 

a data storage step in which the three-dimensional shapes 
measured in the three-dimensional shape measurement step and the 
30 texture images obtained in the texture image photographing step are 
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stored as the registered data, and 

the illumination correction step includes- 

an image generation step in which illumination variation 
images of each registered object in the same position/pose as the target 
5 object in the input image and under various illumination conditions are 
generated by use of the position/pose corrected in the position/pose 
correction step and the three-dimensional shapes and the texture images 
of the object stored in the storage step; and 

an illumination condition estimation step in which an image 
10 that is the most similar to the input image obtained in the 
photographing step is generated as the reference image by use of the 
illumination variation images generated in the image generation step. 

[Claim 40] The image comparison method as claimed in claim 39, 
wherein the illumination correction step further includes an illumination 
15 variation space generation step in which an illumination variation space 
which is spanned by the illumination variation images generated in the 
image generation step is generated before the reference image is 
generated in the illumination condition estimation step, and 

in the illumination condition estimation step, the image that is 
20 the most similar to the input image is generated as the reference image 
from the illumination variation space generated in the illumination 
variation space generation step. 

[Claim 41] The image comparison method as claimed in claim 40, 
wherein: 

25 in the illumination variation space generation step, basis 

vectors of a space that almost accommodates image variation due to the 
illumination variation are generated by conducting the principal 
component analysis to the illumination variation images generated in 
the image generation step, and 

30 in the illumination condition estimation step, correlations are 
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obtained between the input image obtained in the photographing step 
and each of the basis vectors generated in the illumination variation 
space generation step, and the image that is the most similar to the 
input image is generated as the reference image by use of the basis 
5 vectors and based on the correlations. 

[Claim 42] The image comparison method as claimed in claim 31, 
wherein in the position/pose correction step, the position/pose of the 
object is the corrected position/pose of a predetermined position/pose. 

[Claim 43] The image comparison method as claimed in claim 31, 
10 wherein in the position/pose correction step, the position/pose is a 
corrected position/pose to which position/pose parameters of the object 
photographed by the photographing means are input. 

[Claim 44] The image comparison method as claimed in claim 31, 
wherein in the position/pose correction step, the position/pose is a 
15 corrected position/pose of the estimated position/pose of the object 
photographed by the photographing means. 

[Claim 45] The image comparison method as claimed in claim 44, 
wherein: 

the registration step includes: 
20 a three-dimensional shape measurement step in which the 

three-dimensional shape of each object to be registered is measured; 

a reflectance measurement step in which the surface reflectance 

at each position of the three-dimensional shape of the object to be 

registered is measured; 
25 a first feature point extraction step in which feature point 

positions of the registered object are extracted based on the 

three-dimensional shape measured in the three-dimensional shape 

measurement step and the reflectance measured in the reflectance 

measurement step; and 
30 a data storage step in which the three-dimensional shapes 
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measured in the three-dimensional shape measurement step, the 

reflectance measured in the reflectance measurement step and the 

feature point positions extracted in the first feature point extraction step 

are stored as the registered data, and 
5 the position/pose correction step includes: 

a second feature point extraction step in which feature point 

positions same as those stored in the data storage step are extracted 

from the input image obtained in the photographing step as input image 

feature point positions; and 
10 a position/pose calculation step in which the position/pose of the 

object photographed in the photographing means is estimated based on 

the three-dimensional shape and the feature point positions stored in the 

data storage step and the input image feature point positions obtained in 

the second feature point extraction step. 
15 [Claim 46] The image comparison method as claimed in claim 45, 

wherein the illumination correction step includes: 

an image generation step in which images of each object in the 

same position/pose as the object obtained in the photographing step and 

under various illumination conditions are generated as illumination 
20 variation images by use of the position/pose estimated in the 

position/pose determination step and the three-dimensional shape and 

the reflectance stored in the data storage step; and 

an illumination condition estimation step in which an image 

that is the most similar to the input image obtained in the 
25 photographing step is generated as the reference image by use of the 

illumination variation images generated in the image generation step. 
[Claim 47] The image comparison method as claimed in claim 46, 

wherein: 

the illumination correction step further includes an illumination 
30 variation space generation step in which an illumination variation space 
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which is spanned by the illumination variation images generated in the 
image generation step is generated before the reference image is 
generated in the illumination condition estimation step, and 

in the illumination condition estimation step, the image that is 
5 the most similar to the input image is generated as the reference image 
from the illumination variation space generated in the illumination 
variation space generation step. 

[Claim 48] The image comparison method as claimed in claim 47, 
wherein^ 

10 in the illumination variation space generation step, basis 

vectors of a space that almost accommodates image variation due to the 
illumination variation are generated by conducting principal component 
analysis to the illumination variation images generated in the image 
generation step, and 

15 in the illumination condition estimation step, correlations are 

obtained between the input image obtained in the photographing step 
and each of the basis vectors generated in the illumination variation 
space generation step, and the image that is the most similar to the 
input image obtained in the photographing step is generated as the 

20 reference image by use of the basis vectors and based on the correlations. 

[Claim 49] The image comparison method as claimed in one of 
claims 46 to 48, wherein the illumination correction step further includes 
an illumination condition variation step in which various illumination 
conditions are set and generated before illumination variation images 

25 are generated in the image generation step. 

[Claim 50] The image comparison method as claimed in claim 30, 
wherein: 

the registration step includes^ 

a three-dimensional shape measurement step in which the 
30 three-dimensional shape of each object to be registered is measured; 
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a reflectance measurement step in which the surface reflectance 
at each position of the three-dimensional shape of the object to be 
registered is measured; 

an image generation step in which images of each object to be 
5 registered under various illumination conditions are generated as 
illumination variation images; 

an illumination variation space generation step in which an 
illumination variation space which is spanned by the illumination 
variation images generated in the image generation step is generated; 
10 and 

a data storage step in which the illumination variation space 
generated in the illumination variation space generation step is stored as 
the registered data, and 

the comparison step includes ' 
15 a photographing step in which the comparison target object to be 

compared with the registered object registered in the registration step is 
photographed as an input image is obtained; 

an illumination condition estimation step in which an image of 
each object that is the most similar to the input image obtained in the 
20 photographing step is generated as a reference image from the 
illumination variation space stored in the data storage step; 

an image comparison step in which each reference image 
generated in the illumination condition estimation step is compared with 
the input image obtained in the photographing step and thereby an 
25 evaluation value concerning the similarity between the two images is 
calculated; and 

a comparison judgment step in which it is judged whether or not 
each of the registered objects registered in the registration step is the 
target object photographed in the photographing step based on the 
30 evaluation value calculated in the image comparison step. 
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[Claim 51] The image comparison method as claimed in claim 30, 
wherein" 

the registration step includes* 

a three-dimensional shape measurement step for measuring the 
5 three-dimensional shape of each object to be registered; 

a reflectance measurement step for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered; 

an image generation step for generating images of each object to 
10 be registered under various illumination conditions as illumination 
variation images; 

an illumination variation space generation step for generating 
basis vectors of a space that almost accommodates image variation due 
to the illumination variation by conducting principal component analysis 
15 to the illumination variation images generated in the image generation 
step; and 

a data storage step for storing, as the registered data, the basis 
vectors generated by the illumination variation space generation step, 
and 

20 the comparison step includes: 

a photographing step for photographing, as an input image, the 
target object to be compared with the data of registered objects 
registered in the registration step; 

an illumination condition estimation step for obtaining 
25 correlations between the input image obtained by the photographing step 
and each of the basis vectors stored in the data storage step, generating, 
as the reference image, the image that is the most similar to the input 
image by use of the basis vectors and based on the correlations; 

an image comparison step for comparing each reference image 
30 generated by the illumination condition estimation step with the input 
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image obtained by the photographing step and thereby calculating an 
evaluation value concerning the similarity between the two images; and 

a comparison judgment step for judging whether or not each of 
the registered objects registered in the registration step is the object 
5 photographed by the photographing step based on the evaluation value 
calculated by the image comparison step. 

[Claim 52] The image comparison method as claimed in one of 
claims 31 to 51, wherein the three-dimensional shape measurement step 
measures the three-dimensional shapes by reading data or drawings. 
10 [Claim 53] The image comparison method as claimed in one of 

claims 31 to 52, wherein the reflectance measurement step obtains the 
reflectance by reading data or drawings. 

[Claim 54] The image comparison method as claimed in one of 
claims 31 to 53, wherein the photographing step obtains the input 
15 image by scanning a film, a photograph or printed matter. 

[Claim 55] The image comparison method as claimed in one of 
claims 31 to 54, wherein the comparison judgment step judges which 
registered object matches the target object. 

[Claim 56] The image comparison method as claimed in one of 
20 claims 31 to 55, wherein the comparison judgment step searches for one 
or more registered objects that are similar to the target object. 

[Claim 57] The image comparison method as claimed in one of 
claims 30 to 56, wherein the registered objects are automobiles. 

[Claim 58] The image comparison method as claimed in one of 
25 claims 30 to 57, wherein the objects to be registered are human faces. 

[Claim 59] A record medium storing an image comparison program 
executing the processes of- 

a registration process in which three-dimensional data of each 
object to be registered are registered; and 
30 a comparison process in which two-dimensional data of an object 
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to be compared with the data registered in the registration step is 
obtained, and the obtained data is compared with the data registered in 
the registration process. 

[Claim 60] A record medium storing the image comparison program 
5 as claimed in claim 59, wherein the comparison process executes the 
processes of- 

a photographing process in which the comparison target object 
is photographed and thereby an input image is obtained; 

a position/pose correction process in which the position/pose of 
10 the target object obtained in the photographing process is corrected; 

an illumination correction process in which an image of each 
registered object in the same position/pose as the target object in the 
input image and under an illumination condition most similar to that of 
the input image is obtained as a reference image by use of the 
15 position/pose corrected in the position/pose correction process and the 
data registered in the registration process; 

an image comparison process in which each reference image 
generated in the image correction process is compared with the input 
image obtained in the photographing process and thereby an evaluation 
20 value concerning the similarity between the two images is calculated; 
and 

a comparison judgment process in which it is judged whether or 
not each registered object registered in the registration process is the 
object photographed in the photographing process based on the 
25 evaluation value calculated in the image comparison process. 

[Claim 61] A record medium storing the image comparison program 
as claimed in claim 60, wherein the registration process executes the 
processes of 

a three-dimensional shape measurement process in which the 
30 three-dimensional shape of each object to be registered is measured; 
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a reflectance measurement process in which the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered is measured; and 

a data storage process in which the three-dimensional shapes 
5 measured in the three-dimensional shape measurement process and the 
reflectance measured in the reflectance measurement process are stored 
as the registered data. 

[Claim 62] A record medium storing the image comparison program 
as claimed in claim 60, wherein the registration process executes the 
10 processes of 

a three-dimensional shape measurement process in which the 
three-dimensional shape of each object to be registered is measured; 

a image information obtaining process in which the object to be 
registered is photographed and thereby image information of the object 
15 to be registered is obtained; and 

a data storage process in which the three-dimensional shape 
measured in the three-dimensional shape measurement process and the 
image information obtained in the image obtaining process are stored as 
the registered data. 

20 [Claim 63] A record medium storing the image comparison program 

as claimed in claim 60, wherein the registration process executes the 
processes of 

a three-dimensional shape measurement process in which the 
three-dimensional shape of each object to be registered is measured; 
25 an average shape generation process in which an average 

three-dimensional shape is generated as the average of a plurality of the 
three-dimensional shapes measured in the three-dimensional shape 
measurement process when registering a plurality of objects; 

a reflectance measurement process in which the surface 
30 reflectance at each position of the three-dimensional shape of the object 
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to be registered is measured; and 

a data storage process in which the three-dimensional shapes 
measured in the three-dimensional shape measurement process, the 
average three-dimensional shape generated in the average shape 
5 generation process and the reflectance measured in the reflectance 
measurement process are stored as the registered data. 

[Claim 64] A record medium storing the image comparison program 
as claimed in one of claims 61 to 63, wherein the illumination correction 
process executes the processes of 

10 an image generation process in which images of each registered 

object in the same position/pose as the target object photographed by the 
photographing means and under various illumination conditions are 
generated as illumination variation images by use of the position/pose 
corrected in the position/pose correction process and the data registered 

15 in the registration process! and 

an illumination condition estimation process in which an image 
that is the most similar to the input image obtained in the 
photographing process is generated as the reference image by use of the 
illumination variation images generated in the image generation process. 

20 [Claim 65] A record medium storing the image comparison program 

as claimed in claim 64, wherein- 

the illumination correction process further includes an 
illumination variation space generation process in which an illumination 
variation space which is spanned by the illumination variation images 

25 generated in the image generation process is generated, and 

in the illumination condition estimation process, the image that 
is the most similar to the input image is generated as the reference 
image from the illumination variation space generated in the 
illumination variation space generation process. 

30 [Claim 66] A record medium storing the image comparison program 
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as claimed in claim 65, wherein: 

in the illumination variation space generation process, basis 
vectors of a space that almost accommodates image variation due to the 
illumination variation are generated by conducting principal component 
5 analysis to the illumination variation images generated in the image 
generation process, and 

in the illumination condition estimation process, correlations 
are obtained between the input image obtained in the photographing 
process and each of the basis vectors generated in the illumination 
10 variation space generation process, and the image that is the most 
similar to the input image is generated as the reference image by use of 
the basis vectors and based on the correlations. 

[Claim 67] A record medium storing the image comparison program 
as claimed in one of claims 64 to 66, wherein the illumination correction 
15 process further includes an illumination condition variation process in 
which various illumination conditions are set and generated to be used 
for generating illumination variation images before the illumination 
variation images are generated in the image generation process. 

[Claim 68] A record medium storing the image comparison program 
20 as claimed in claim 60, wherein^ 

the registration process includes: 

a three-dimensional shape measurement process in which the 
three-dimensional shape of each object to be registered is measured; 

a texture image photographing process in which the object to be 
25 registered is photographed under various illumination conditions and 
thereby texture images are obtained; and 

a data storage process in which the three-dimensional shapes 
measured in the three-dimensional shape measurement process and the 
texture images obtained in the texture image photographing process are 
30 stored as the registered data, and 
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the illumination correction process includes- 

an image generation process in which illumination variation 
images of each registered object in the same position/pose as the target 
object in the input image and under various illumination conditions are 
5 generated by use of the position/pose corrected in the position/pose 
correction process and the three-dimensional shapes and the texture 
images of the object stored in the storage process; and 

an illumination condition estimation process in which an image 
that is the most similar to the input image obtained in the 
10 photographing process is generated as the reference image by use of the 
illumination variation images generated in the image generation process. 

[Claim 69] A record medium storing the image comparison program 
as claimed in claim 68, wherein the illumination correction process 
further includes an illumination variation space generation process in 
15 which an illumination variation space which is spanned by the 
illumination variation images generated in the image generation process 
is generated before the reference image is generated in the illumination 
condition estimation process, and 

in the illumination condition estimation process, the image that 
20 is the most similar to the input image is generated as the reference 
image from the illumination variation space generated in the 
illumination variation space generation process. 

[Claim 70] A record medium storing the image comparison program 
as claimed in claim 69, wherein: 
25 in the illumination variation space generation process, basis 

vectors of a space that almost accommodates image variation due to the 
illumination variation are generated by conducting the principal 
component analysis to the illumination variation images generated in 
the image generation process, and 
30 in the illumination condition estimation process, correlations 
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aire obtained between the input image obtained in the photographing 
process and each of the basis vectors generated in the illumination 
variation space generation process, and the image that is the most 
similar to the input image is generated as the reference image by use of 
5 the basis vectors and based on the correlations. 

[Claim 71] A record medium storing the image comparison program 
as claimed in claim 60, wherein in the position/pose correction process, 
the position/pose of the object is the corrected position/pose of a 
predetermined position/pose. 
10 [Claim 72] A record medium storing the image comparison program 

as claimed in claim 60, wherein in the position/pose correction process, 
the position/pose is a corrected position/pose to which position/pose 
parameters of the object photographed by the photographing means are 
input. 

15 [Claim 73] A record medium storing the image comparison program 

as claimed in claim 60, wherein in the position/pose correction process, 
the position/pose is a corrected position/pose of the estimated 
position/pose of the object photographed by the photographing means. 
[Claim 74] A record medium storing the image comparison program 
20 as claimed in claim 73, wherein: 

the registration process includes^ 

a three-dimensional shape measurement process in which the 
three-dimensional shape of each object to be registered is measured; 

a reflectance measurement process in which the surface 
25 reflectance at each position of the three-dimensional shape of the object 
to be registered is measured; 

a first feature point extraction process in which feature point 
positions of the registered object are extracted based on the 
three-dimensional shape measured in the three-dimensional shape 
30 measurement process and the reflectance measured in the reflectance 
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measurement process; and 

a data storage process in which the three-dimensional shapes 
measured in the three-dimensional shape measurement process, the 
reflectance measured in the reflectance measurement process and the 
5 feature point positions extracted in the first feature point extraction 
process are stored as the registered data, and 

the position/pose correction process includes: 

a second feature point extraction process in which feature point 
positions same as those stored in the data storage process are extracted 
10 from the input image obtained in the photographing process as input 
image feature point positions; and 

a position/pose calculation process in which the position/pose of 
the object photographed in the photographing means is estimated based 
on the three-dimensional shape and the feature point positions stored in 
15 the data storage process and the input image feature point positions 
obtained in the second feature point extraction process. 

[Claim 75] A record medium storing the image comparison program 
as claimed in claim 74, wherein the illumination correction process 
includes : 

20 an image generation process in which images of each object in 

the same position/pose as the object obtained in the photographing 
process and under various illumination conditions are generated as 
illumination variation images by use of the position/pose estimated in 
the position/pose determination process and the three-dimensional shape 

25 and the reflectance stored in the data storage process; and 

an illumination condition estimation process in which an image 
that is the most similar to the input image obtained in the 
photographing process is generated as the reference image by use of the 
illumination variation images generated in the image generation process. 

30 [Claim 76] A record medium storing the image comparison program 
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as claimed in claim 75, wherein: 

the illumination correction process further includes an 
illumination variation space generation process in which an illumination 
variation space which is spanned by the illumination variation images 
5 generated in the image generation process is generated before the 
reference image is generated in the illumination condition estimation 
process, and 

in the illumination condition estimation process, the image that 
is the most similar to the input image is generated as the reference 
10 image from the illumination variation space generated in the 
illumination variation space generation process. 

[Claim 77] A record medium storing the image comparison program 
as claimed in claim 76, wherein: 

in the illumination variation space generation process, basis 
15 vectors of a space that almost accommodates image variation due to the 
illumination variation are generated by conducting principal component 
analysis to the illumination variation images generated in the image 
generation process, and 

in the illumination condition estimation process, correlations 
20 are obtained between the input image obtained in the photographing 
process and each of the basis vectors generated in the illumination 
variation space generation process, and the image that is the most 
similar to the input image obtained in the photographing process is 
generated as the reference image by use of the basis vectors and based 
25 on the correlations. 

[Claim 78] A record medium storing the image comparison program 
as claimed in one of claims 75 to 77, wherein the illumination correction 
process further includes an illumination condition variation process in 
which various illumination conditions are set and generated before 
30 illumination variation images are generated in the image generation 
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process. 

[Claim 79] A record medium storing the image comparison program 
as claimed in claim 59, wherein: 

the registration process includes-' 
5 a three-dimensional shape measurement process in which the 

three-dimensional shape of each object to be registered is measured; 

a reflectance measurement process in which the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered is measured; 
10 an image generation process in which images of each object to be 

registered under various illumination conditions are generated as 
illumination variation images; 

an illumination variation space generation process in which an 
illumination variation space which is spanned by the illumination 
15 variation images generated in the image generation process is generated; 
and 

a data storage process in which the illumination variation space 
generated in the illumination variation space generation process is 
stored as the registered data, and 
20 the comparison process includes: 

a photographing process in which the comparison target object 
to be compared with the registered object registered in the registration 
process is photographed as an input image is obtained; 

an illumination condition estimation process in which an image 
25 of each object that is the most similar to the input image obtained in the 
photographing process is generated as a reference image from the 
illumination variation space stored in the data storage process; 

an image comparison process in which each reference image 
generated in the illumination condition estimation process is compared 
30 with the input image obtained in the photographing process and thereby 
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an evaluation value concerning the similarity between the two images is 
calculated; and 

a comparison judgment process in which it is judged whether or 
not each of the registered objects registered in the registration process is 
5 the target object photographed in the photographing process based on 
the evaluation value calculated in the image comparison process. 

[Claim 80] A record medium storing the image comparison program 
as claimed in claim 59, wherein- 

the registration process includes: 
10 a three-dimensional shape measurement process for measuring 

the three-dimensional shape of each object to be registered; 

a reflectance measurement process for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered; 

15 an image generation process for generating images of each 

object to be registered under various illumination conditions as 
illumination variation images; 

an illumination variation space generation process for 
generating basis vectors of a space that almost accommodates image 
20 variation due to the illumination variation by conducting principal 
component analysis to the illumination variation images generated in 
the image generation process; and 

a data storage process for storing, as the registered data, the 
basis vectors generated by the illumination variation space generation 
25 process, and 

the comparison process includes: 

a photographing process for photographing, as an input image, 
the target object to be compared with the data of registered objects 
registered in the registration process! 
30 an illumination condition estimation process for obtaining 
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correlations between the input image obtained by the photographing 
process and each of the basis vectors stored in the data storage process, 
generating, as the reference image, the image that is the most similar to 
the input image by use of the basis vectors and based on the correlations; 
5 an image comparison process for comparing each reference 

image generated by the illumination condition estimation process with 
the input image obtained by the photographing process and thereby 
calculating an evaluation value concerning the similarity between the 
two images; and 

10 a comparison judgment process for judging whether or not each 

of the registered objects registered in the registration process is the 
object photographed by the photographing process based on the 
evaluation value calculated by the image comparison process. 

[Claim 81] A record medium storing the image comparison program 

15 as claimed in one of claims 60 to 80, wherein the three-dimensional 
shape measurement process measures the three-dimensional shapes by 
reading data or drawings. 

[Claim 82] A record medium storing the image comparison program 
as claimed in one of claims 60 to 81, wherein the reflectance 

20 measurement process obtains the reflectance by reading data or 
drawings. 

[Claim 83] A record medium storing the image comparison program 
as claimed in one of claims 60 to 82, wherein the photographing 
process obtains the input image by scanning a film, a photograph or 
25 printed matter. 

[Claim 84] A record medium storing the image comparison program 
as claimed in one of claims 60 to 83, wherein the comparison judgment 
process judges which registered object matches the target object. 

[Claim 85] A record medium storing the image comparison program 
30 as claimed in one of claims 60 to 84, wherein the comparison judgment 
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process searches for one or more registered objects that are similar to the 
target object. 

[Claim 86] A record medium storing the image comparison program 
as claimed in one of claims 59 to 85, wherein the registered objects are 
5 automobiles. 

[Claim 87] A record medium storing the image comparison program 
as claimed in one of claims 59 to 86, wherein the objects to be registered 
are human faces. 

[Detailed Description of the Invention] 
10 [0001] 

[Field of the Invention] 

The present invention relates to an image comparison device 
and an image comparison method for comparing a target object with 
registered objects by use of images, and in particular, to an image 
15 comparison device and an image comparison method capable of 
conducting the comparison correctly withstanding variations of shooting 
conditions such as position/pose of the target object in the image and 
illumination conditions, by preliminarily registering three-dimensional 
shape information, surface reflectance information, color information etc. 
20 of the registered objects. 
[0002] 
[Prior Art] 

As shown in Fig. 5, image comparison techniques are such 
techniques that an object existing in the three-dimensional space in a 

25 random position/pose is photographed to acquire one or more input 
images by use of a photographing device such as camera or video camera. 
Thereafter, the input images of the target object are compared with 
images of objects which have preliminarily been registered. The image 
comparison process is generally composed of two steps: a registration 

30 step for registering and storing the compared objects; and a comparison 
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step for comparing the input image of the target object with the 
registered data of the registered objects and thereby judging what the 
target object is. In each step, the photographed image is used as a 
two-dimensional image without being processed, or is used after being 
5 converted to a three-dimensional shape. In the following, conventional 
image comparison techniques will be explained in detail with reference 
to documents. 
[0003] 
(Prior Art l) 

10 As an example of an image comparison technique in which 

two-dimensional images are registered beforehand and a 
two-dimensional image is used as an input image, there is disclosed a 
"face image comparison device" in Japanese Patent No. 2872776. This 
prior art has been designed to conduct image comparison of human faces 

15 and has such configuration as shown in Figs. 1 and 3. In the 
registration step, two-dimensional images are obtained by use of a 
camera 11, and the obtained two-dimensional images are stored in a 
storage means 12. In the comparison step, a two-dimensional face 
image is obtained as the input image by use of a camera 13. A 

20 normalization means 14 extracts feature points being eyes, nose, etc. to 
be used as the criteria for judging pose, size, etc. from the input image by 
means of image processing. A normalized image which is normalized in 
accordance with two-dimensional position and size on an image is output 
by use of the feature points as coordinates position. Thereafter, an 

25 image comparison means 15 compares registered images reads out from 
the storage means 12 with the normalized image by means of pattern 
recognition, and outputs the result of the comparison. 
[0004] 
(Prior Art 2) 

30 As an example of a conventional comparison techniques using 
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three-dimensional shapes, there is disclosed a "personal identification 
device" in Japanese Patent Application Laid-Open No.HEI9-259271. In 
this prior art, there is configured as shown in Fig. 14. In the 
registration step, a three-dimensional shape color measurement means 
5 21 measures three-dimensional shape and color of each comparison 
target object, and the three-dimensional shape information and the color 
information are stored in a storage means 22. In the comparison step, a 
three-dimensional shape color measurement means 23 measures 
three-dimensional shape and color of a target object as input data. A 

10 translation/rotation means 24 translates the input data so that the 
barycenter of the input data will be on the barycenter of the registered 
data. Subsequently, the translation/rotation section 24 rotates the 
translated input data in various ways, and a minimum error calculation 
means 25 obtains the error between the rotated input data and the 

15 registered data. The minimum error calculation means 25 adjusts the 
three-dimensional position and pose of the input data. Thereafter, the 
minimum error calculation section 25 compares the corrected data with 
the registered data. 
[0005] 

20 (Prior Art 3) 

A "personal identification device" disclosed in Japanese Patent 
Application Laid-Open No.HEI6-168317 uses two-dimensional images 
both in the registration step and the comparison step, and the 
configuration thereof is as shown in Fig. 15. In the registration step, 

25 two-dimensional images are obtained by use of a camera 41. A feature 
extraction means 42 extracts feature points where variations in pixel 
intensity are large and stores position data of the feature points in a 
storage means 43. In the comparison step, a two-dimensional image is 
obtained as an input image by use of a camera 44. Subsequently, a 

30 feature extraction section 45 extracts feature points where variations in 
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pixel intensity are large from the two-dimensional input image and 
outputs position data of the feature points. Subsequently, the 
comparison means 47 conducts the personal identification by comparing 
the registered feature point position and the feature position of the input 
5 image. Here, in order to compensate for and absorb variations in 
position/pose of the target object, the feature point position data output 
by the feature extraction means 45 is normalized by a position/pose 
normalization means 46 by use of a three-dimensional shape model of a 
standard object which has been prepared in the position/pose 
10 normalization means 46. 
[0006] 
(Prior Art 4) 

A technique disclosed in a document: Hiroshi Murase and Shree 
K. Nayer "Visual Learning and Recognition of three-dimensional objects 

15 from Appearance", Int. J. Computer Vision, vol.14, pp. 5-24 (1995) uses 
only two-dimensional images both in the registration step and the 
comparison step and executes correction not only for variations in 
position/pose but also for variations due to illumination conditions. Fig. 
16 shows the configuration of the conventional art. In the registration 

20 step, a large set of sample images of each object are obtained by a 
photographing means 71 by automatically varying pose of the object and 
illumination conditions so that all feasible poses and illumination 
conditions can be covered. A manifold calculation means 72 obtains 
basis images capable of expressing the variations of the sample images, 

25 by means of principal component analysis. Subsequently, the manifold 
calculation section 72 generates a feature space which is spanned by the 
basis images, obtains a trajectory of the sample images in the feature 
space as a manifold, and stores the obtained manifold in a storage 
section 73. In the comparison step, a two-dimensional image of a target 

30 object is obtained by a camera 74 as an input image. A distance 
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calculation means 75 calculates the distance between the input image 
and the manifold in the feature space and executes the comparison using 
the distance as a yardstick. Thereby, the comparison by use of the input 
image, which can be obtained under various illumination conditions and 
5 poses, is made possible. 
[0007] 

(Prior Art 5) 

Changes occurring to a two-dimensional image due to variations 
of illumination conditions when the position/pose of the object is fixed 

10 has been analyzed in detail in a document" Peter N. Belhumeur and 
David J. Kriegman "What is the Set of Images of an Object Under All 
Possible Illumination Conditions ?", Int. J. Computer Vision, vol.28, 
pp. 245-260 (1998). In the case where the position and pose of the object 
are fixed, an image obtained under an arbitrary illumination condition 

15 can be resolved into images each of which is obtained under a point 
source of light. Therefore, an image obtained under an arbitrary 
number of point sources of light can be expressed as a linear combination 
of images each of which is obtained under a point source of light, in 
which the intensity of each point source is used as each coefficient of the 

20 linear combination. Based on the above analysis, a method called 
"illumination subspace method" which is shown in Fig. 17 has been 
proposed. 
[0008] 

Referring to Fig. 17, a photographing means 51 selects and sets 
25 three or more illumination conditions so that the number of pixels in 
shadow will be as small as possible, and obtains images under the three 
or more illumination conditions. A normal line calculation means 52 
obtains reflectance vectors on the surface of the object, with regard to 
each pixel in the images, by means of principal component analysis. An 
30 image generation means 53 successively generates images which are 
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called "extreme rays" and stores the generated extreme rays in a storage 
means 54. 
[0009] 

In the comparison step, a two-dimensional image of a target 
5 object is obtained by use of a camera 55 as an input image. When the 
reflectance property of the surface of an object is perfect scattering and 
the surface is convex, an image of the object under an arbitrary 
illumination condition can be expressed by a linear combination of the 
extreme rays with positive coefficients, and therefore, an illumination 
10 correction means 56 generates a reference image being an image of the 
registered object under the same illumination condition as in the input 
image by use of a linear combination of the extreme rays. The 
coefficients of the linear combination are determined by use of the 
nonnegative least-squares method. An image comparison means 57 
15 compares the input image with the reference image by calculating the 
similarity between the two images. 
[0010] 
(Prior Art 6) 

In a technique disclosed in a document: A. S. Georghiades et al. 
20 "Illumination Cones for Recognition Under Variable Lighting: Faces", 
Proc. IEEE Int. Conf. CVPR, pp. 52-58 (1998), when the extreme rays are 
obtained according to the "illumination subspace method", pixels that 
should be in shadow are found out based on the three-dimensional shape 
of the object by use of techniques of computer graphics such as ray 
25 tracing, and the pixels are shadowed. The technique of this document 
aims to apply the "illumination subspace method" to objects having 
concavities. 
[0011] 
(Prior Art 7) 

30 The aforementioned document: Peter N. Belhumeur and David J. 
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Kriegman "What is the Set of Images of an Object Under All Possible 
Illumination Conditions ?", Int. J. Computer Vision, vol.28, pp. 245-260 
(1998), also proposes a "sampling method" which is shown in Fig. 18. It 
takes much time to calculate all the extreme rays as in the "illumination 
5 subspace method", and therefore, a photographing means 61 in the 
registration step selects and sets an appropriate number of illumination 
directions so that the angles ( d , <t> ) shown in Fig. 4 will cover the 
sphere at almost even intervals for example, and obtains images. The 
images obtained as above is substituted for the extreme rays. In the 
10 comparison step, the illumination correction is executed by means of the 
nonnegative least-squares method and thereafter the object recognition 
is executed. 
[0012] 

[Problems that the Invention is to Solve] 

15 However, the target object in front of a photographing device 

being camera, video camera, etc., generally moves in the 
three-dimensional space (parallel translation, rotation, etc.) unless the 
position/pose of the target object is fixed or adjusted. Further, the 
illumination conditions vary every moment when the photographing of 

20 the target object is done outdoors. Therefore, the appearance of the 
two-dimensional input image of the target object is necessitated to vary 
widely. The conventional techniques could not compensate for and 
absorb enough the variations in the position/pose of the target object and 
the illumination conditions, thereby the application of the conventional 

25 image comparison techniques used to be confined within narrow limits. 
In the following, the problems in the conventional techniques of the 
above documents will be described in detail. 
[0013] 

The "face image comparison device" of Japanese Patent 
30 No.2872776 in accordance with the Prior Art 1 employing the simple 
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image comparison between two-dimensional images can not cope with 
variations of appearance in two-dimensional images caused by changes 
of the pose of the target object being three-dimensional rotation, etc., 
illumination conditions, etc. Therefore, the application of the technique 
5 of the document No.l is extremely limited. 
[0014] 

The image comparison technique employed by the "personal 
identification device" of Japanese Patent Application Laid-Open 
No.HEI9-259271 requires three-dimensional shapes not only in the 

10 registration step but also in the comparison step. A three-dimensional 
shape measurement device becomes necessary on each comparison, and 
thus, the cost for the comparison is necessitated to be high, especially 
when the measurement of the target object has to be done away from the 
place where the registration step has performed. Further, for the 

15 measurement of the three-dimensional shape, each object has to stand 
still until the measurement is completed, and a darkroom etc. becomes 
necessary for obtaining precise data. Therefore, the application of the 
technique of the document No. 2 is also limited. 
[0015] 

20 The technique employed by the "personal identification device" 

of Japanese Patent Application Laid-Open No. HEI6- 168317, which 
detects pixel positions where variations in intensity are large, is effective 
for comparing objects such as blocks having large three-dimensional 
curvatures, black marks on a white board having steep change of 

25 reflectance, etc. However, the technique is not suitable for comparison 
of human faces as mentioned in the document, which means that stable 
and reliable detection of the feature points is difficult. The document 
also refers to a correction of pose by use of a standard three-dimensional 
shape of target objects, however, the method can not be employed unless 

30 the objects are similar in shape. 
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[0016] 

In the technique disclosed in Hiroshi Murase and Shree K. 
Nayer "Visual Learning and Recognition of three-dimensional objects 
from Appearance", Int. J. Computer Vision, vol.14, pp. 5*24 (1995), 
5 enormous amount of two-dimensional sample images have to be gathered, 
taking the possibilities of various illumination conditions (two or more 
point sources, extra lights other than point sources, etc.) into 
consideration. Further, the shape of the manifold in the feature space is 
not defined in the document at all, therefore, it is difficult to find 
10 appropriate sets of parameters when the distance between the input 
image and the manifold is calculated in the feature space. Therefore, 
large amounts of calculations are necessary for implementing the 
technique of the document. 
[0017] 

15 The "illumination subspace method" and the "sampling method" 

disclosed in Peter N. Belhumeur and David J. Kriegman "What is the Set 
of Images of an Object Under All Possible Illumination Conditions ?", Int. 
J. Computer Vision, vol.28, pp. 245-260 (1998) in accordance with the 
Prior Arts 5 to 7 require images of the registered object illuminated from 

20 many directions. A specially designed lighting unit has to be used in 
the registration step, and precise setting of the illumination conditions is 
difficult from the viewpoints of placement and setting of equipment. 
[0018] 

Further, in the "illumination subspace method" and the 
25 "sampling method", if the position or pose of an object changed, images of 
the object in the new pose have to be obtained again under a lot of 
illumination conditions, and the calculations have to be done all over 
again. Therefore, in the registration step, a huge amount of images of 
the registered object in various feasible poses have to be obtained under 
30 a lot of illumination conditions, taking much time and manpower. If the 
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target object is photographed in the comparison step in a different pose, 
the comparison becomes impossible. 
[0019] 

Moreover, the "illumination subspace method" requires large 
5 amounts of calculation for obtaining the extreme rays, depending on the 
complexity of the shape. According to the document: Peter N. 
Belhumeur and David J. Kriegman "What is the Set of Images of an 
Object Under All Possible Illumination Conditions ?", Int. J. Computer 
Vision, vol.28, pp. 245*260 (1998), when there are M independent normal 
10 vectors on the surface of the object, the number of the extreme rays 
amounts to M(M-l) at the maximum. Therefore, an enormous amount 
of extreme rays have to be calculated unless the shape of the object is 
very simple as a block. Therefore, it is very difficult to calculate all the 
extreme rays for ordinary objects having complex shapes. Further, the 
15 method can not be directly applied to cases where the object has a 
concavity, and thereby, a cast shadow occurs. 
[0020] 

In the "illumination subspace method" and the "sampling 
method", the amount of calculations in the nonnegative least-squares 
20 method also tends to be enormous depending on the number of extreme 
rays. In addition to the above problems, the sampling method also has 
a problem that it is not clear how many basis images are necessary for 
obtaining satisfactory result and performance. 
[0021] 

25 Further, the "illumination subspace method" and the "sampling 

method" have been constructed assuming that the surface reflectance 
property of each object is perfect scattering. Therefore, the methods can 
not be applied directly to cases where mirror reflection exists or cases 
where the surface reflectance property is diffusion reflection but not 

30 perfect scattering. The majority of ordinary objects do not exhibit the 
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surface perfect scattering. 
[0022] 

It is therefore the primary object of the present invention to 
provide an image comparison device and an image comparison method by 
5 which the comparison step can be conducted by use of a two-dimensional 
image of the target object obtained by use of an ordinary camera, without 
needing a three-dimensional shape of the target object as the input data. 
[0023] 

Another object of the present invention is to provide an image 
10 comparison device and an image comparison method by which the 
comparison of objects can be conducted correctly compensating for and 
absorbing the three-dimensional changes of the position/pose of the 
target object in the input image, necessary data of the registered objects 
can be gathered in the registration step easily, and the correction of 
15 illumination conditions can be conducted with regard to the input image 
which can be obtained under various illumination conditions at high 
processing speed. 
[0024] 

[Means of Solving the Problems] 
20 In accordance with a first aspect of the present invention, there 

is provided an image comparison device comprising: 

a registration means for registering data of a registration target 
object as three-dimensional data; and 

a comparison means for obtaining, as two-dimensional data, 
25 data of an object being registered in the registration means and data of 
an object to be compared and comparing the data of the registered object 
in the registration means and obtained data. 
[0025] 

In accordance with a second aspect of the present invention, in 
30 the first aspect, the comparison means includes: 
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a photographing means for photographing, as an input image, 
the target object to be compared with the data of registered objects 
registered in the registration means; 

a position/pose correction means for correcting the position/pose 
5 of the target object obtained by the photographing means; 

an illumination correction means for generating, as a reference 
image, an image in the same position/pose as the input image, which is 
photographed by the photographing means, and under an illumination 
condition most similar to that of the input image by use of the 
10 position/pose corrected by the position/pose correction means and the 
data registered in the registration means; 

an image comparison means for comparing the reference image 
generated by the illumination correction means with the input image 
obtained by the photographing means and thereby calculating an 
15 evaluation value concerning the similarity between the two images; and 

a comparison judgment means for judging whether or not the 
target object photographed by the photographing means is the registered 
object registered in the registration means based on the evaluation value 
calculated by the image comparison means. 
20 [0026] 

In accordance with a third aspect of the present invention, in 
the second aspect, the registration means includes^ 

a three-dimensional shape measurement means for measuring 
the three-dimensional shape of each object to be registered; 
25 a reflectance measurement means for measuring the surface 

reflectance at each position on the three-dimensional shape of the object 
to be registered; and 

a data storage means for storing, as the registered data, the 
three-dimensional shapes measured by the three-dimensional shape 
30 measurement means and the reflectance measured by the reflectance 
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measurement means. 
[0027] 

In accordance with a fourth aspect of the present invention, in 
the second aspect, the registration means includes: 
5 a three-dimensional shape measurement means for measuring 

the three-dimensional shape of each object to be registered; 

an image information obtaining means for obtaining image 
information of the object by photographing the object to be registered; 
and 

10 a data storage means for storing, as the registered data, the 

three-dimensional shapes measured by the three-dimensional shape 
measurement means and the image information obtained by the image 
information obtaining means. 
[0028] 

15 In accordance with a fifth aspect of the present invention, in the 

second aspect, the registration means includes^ 

a three-dimensional shape measurement means for measuring 
the three-dimensional shape of each object to be registered; 

an average shape generation means for generating an average 
20 three-dimensional shape of the three-dimensional shapes measured by 
the three-dimensional shape measurement means when registering a 
plurality of objects; 

a reflectance measurement means for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 
25 to be registered; and 

a data storage means for storing, as the registered data, the 
three-dimensional shape measured by the three-dimensional shape 
measurement means, the average three-dimensional shape generated by 
the average shape generation means and the reflectance measured by 
30 the reflectance measurement means. 
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[0029] 

In accordance with a sixth aspect of the present invention, in 
one of the third to fifth aspects, the illumination correction means 
includes- 

5 an image generation means for generating, as illumination 

variation images, images of each object in the same position/pose as the 
object photographed by the photographing means and under various 
illumination conditions by use of the position/pose corrected by the 
position/pose correction means and the registered data of objects in the 
10 registration means; and 

an illumination condition estimation means for generating an 
image that is the most similar to the input image obtained by the 
photographing means by use of the illumination variation images 
generated by the image generation means and outputting the generated 
15 image to the image comparison means as the reference image. 
[0030] 

In accordance with a seventh aspect of the present invention, in 
the sixth aspect, the illumination correction means further includes an 
illumination variation space generation means for generating an 
20 illumination variation space which is spanned by the illumination 
variation images generated by the image generation means, and 

the illumination condition estimation means generates the 
image that is the most similar to the input image from the illumination 
variation space generated by the illumination variation space generation 
25 means and outputs the generated image to the image comparison means 
as the reference image. 
[0031] 

In accordance with a eighth aspect of the present invention, in 
the seventh aspect, 

30 the illumination variation space generation means generates 
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basis vectors of a space that almost accommodates image variation due 
to the illumination variation by conducting principal component analysis 
to the illumination variation images generated by the image generation 
means, and 

5 the illumination condition estimation means obtains 

correlations between the input image obtained by the photographing 
means and each of the basis vectors generated by the illumination 
variation space generation means, generates the image that is the most 
similar to the input image by use of the basis vectors and based on the 
10 correlations, and outputs the generated image to the image comparison 
means as the reference image. 
[0032] 

In accordance with a ninth aspect, in one of the sixth to eighth 
aspects, the illumination correction means further includes an 
15 illumination condition variation means for setting various illumination 
conditions and outputting the illumination conditions to the image 
generation means. 
[0033] 

In accordance with a tenth aspect of the present invention, in 
20 the second aspect, 

the registration means includes - 

a three-dimensional shape measurement means for measuring 
the three-dimensional shape of each registered object; 

a texture image photographing means for photographing the 
25 object to be registered under various illumination conditions and thereby 
obtaining texture images of the registered object; and 

a data storage means for storing the three-dimensional shapes 
measured by the three-dimensional shape measurement means and the 
texture images obtained by the texture image photographing means as 
30 the registered data, and 



49 

the illumination correction means includes" 

an image generation means for generating illumination 
variation images of each object in the same position/pose as the object 
photographed by the photographing means by use of the position/pose 
5 determined by the position/pose determination means and the 
three-dimensional shapes and the texture images of the object registered 
in the registration means; and 

an illumination condition estimation means for generating an 
image that is the most similar to the input image obtained by the 
10 photographing means by use of the illumination variation images 
generated by the image generation means and outputting the generated 
image to the image comparison means as the reference image. 
[0034] 

In accordance with a eleventh aspect of the present invention, in 
15 the tenth aspect, wherein the illumination correction means further 
includes an illumination variation space generation means for 
generating an illumination variation space which is spanned by the 
illumination variation images generated by the image generation means, 
and 

20 the illumination condition estimation means generates the 

image that is the most similar to the input image from the illumination 
variation space generated by the illumination variation space generation 
means and outputs the generated image to the image comparison means 
as the reference image. 

25 [0035] 

In accordance with a twelfth aspect of the present invention, in 
the eleventh aspect, 

the illumination variation space generation means generates 
basis vectors of a space that almost accommodates image variation due 
30 to the illumination variation by conducting the principal component 
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analysis to the illumination variation images generated by the image 
generation means, and 

the illumination condition estimation means obtains 
correlations between the input image obtained by the photographing 
5 means and each of the basis vectors generated by the illumination 
variation space generation means, generates the image that is the most 
similar to the input image by use of the basis vectors and based on the 
correlations, and outputs the generated image to the image comparison 
means as the reference image. 
10 [0036] 

In accordance with a thirteenth aspect of the present invention, 
in the second aspect, the position/pose correction means outputs a 
predetermined position/pose to the illumination correction means. 
[0037] 

15 In accordance with a fourteenth aspect of the present invention, 

in the second aspect, the position/pose correction means receives inputs 
concerning position/pose parameters of the object in the input image 
obtained by the photographing means, and outputs the received 
position/pose to the illumination correction means. 

20 [0038] 

In accordance with a fifteenth aspect, in the second aspect, the 
position/pose correction means estimates the position/pose of the object 
in the input image obtained by the photographing means, and outputs 
the estimated position/pose to the illumination correction means. 
25 [0039] 

In accordance with a sixteenth aspect of the present invention, 
in the fifteenth aspect, 

the registration means includes* 

a three-dimensional shape measurement means for measuring 
30 the three-dimensional shape of each object to be registered; 
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a reflectance measurement means for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered; 

a first feature point extraction means for extracting feature 
5 point positions based on the three-dimensional shape measured by the 
three-dimensional shape measurement means and the reflectance 
measured by the reflectance measurement means; and 

a data storage means for storing the three-dimensional shapes 
measured by the three-dimensional shape measurement means, s the 
10 reflectance measured by the reflectance measurement means and the 
feature point positions obtained by the first feature point extraction 
means as the registered data, and 

the position/pose correction means includes - 

a second feature point extraction means for extracting, from the 
15 input image obtained by the photographing means, input image feature 

point positions being the same feature point positions as those stored in 

the storage means; and 

a position/pose calculation means for estimating the 

position/pose of the object photographed by the photographing means 
20 based on the three-dimensional shape and the feature point positions 

stored in the data storage means and the input image feature point 

positions extracted by the second feature point extraction means and 

outputting the estimated position/pose to the illumination correction 

means. 
25 [0040] 

In accordance with a seventeenth aspect of the present 
invention, in the sixteenth aspect, the illumination correction means 
includes: 

an image generation means for generating images of each object 
30 in the same position/pose as the object photographed by the 
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photographing means and under various illumination conditions as 
illumination variation images by use of the position/pose estimated by 
the position/pose correction means and the three-dimensional shape and 
the reflectance stored in the data storage means; and 
5 an illumination condition estimation means for generating an 

image that is the most similar to the input image obtained by the 
photographing means by use of the illumination variation images 
generated by the image generation means and outputting the generated 
image to the image comparison means as the reference image. 
10 [0041] 

In accordance with an eighteenth aspect of the present invention, 
in the seventeenth aspect, 

the illumination correction means further includes an 
illumination variation space generation means for generating an 
15 illumination variation space which is spanned by the illumination 
variation images generated by the image generation means, and 

the illumination condition estimation means generates the 
image that is the most similar to the input image from the illumination 
variation space generated by the illumination variation space generation 
20 means and outputs the generated image to the image comparison means 
as the reference image. 
[0042] 

In accordance with a nineteenth aspect of the present invention, 
in the eighteenth aspect, 
25 the illumination variation space generation means generates 

basis vectors of a space that almost accommodates image variation due 
to the illumination variation by conducting principal component analysis 
to the illumination variation images generated by the image generation 
means, and 

30 the illumination condition estimation means obtains 
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correlations between the input image obtained by the photographing 
means and each of the basis vectors generated by the illumination 
variation space generation means, generates the image that is the most 
similar to the input image photographed by the photographing means by 
5 use of the basis vectors and based on the correlations, and outputs the 
generated image to the image comparison means as the reference image. 
[0043] 

In accordance with a twentieth aspect of the present invention, 
in one of the seventeenth to nineteenth aspects, the illumination 
10 correction means further includes an illumination condition variation 
means for setting various illumination conditions and outputting the 
illumination conditions to the image generation means. 
[0044] 

In accordance with a twenty-first aspect of the present invention, 
15 in the first aspect, 

the registration means includes^ 

a three-dimensional shape measurement means for measuring 
the three-dimensional shape of each object to be registered; 

a reflectance measurement means for measuring the surface 
20 reflectance at each position of the three-dimensional shape of the object 
to be registered; 

an image generation means for generating images of each object 
to be registered under various illumination conditions as illumination 
variation images; 

25 an illumination variation space generation means for generating 

an illumination variation space which is spanned by the illumination 
variation images generated by the image generation means; and 

a data storage means for storing the illumination variation 
space generated by the illumination variation space generation means as 

30 the registered data, and 
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the comparison means includes- 

a photographing means for photographing, as an input image, 
the target object to be compared with the data of registered objects 
registered in the registration means; 
5 an illumination condition estimation means for generating, as a 

reference image, the image that is the most similar to the input image 
from the illumination variation space stored in the data storage means; 

an image comparison means for comparing each reference image 
generated by the illumination condition estimation means with the input 
10 image obtained by the photographing means and thereby calculating an 
evaluation value concerning the similarity between the two images; and 

a comparison judgment means for judging whether or not each 
of the registered objects registered in the registration means is the object 
photographed by the photographing means based on the evaluation value 
15 calculated by the image comparison means. 
[0045] 

In accordance with a twenty-second aspect of the present 
invention, in the first aspect, 

the registration means includes - 
20 a three-dimensional shape measurement means for measuring 

the three-dimensional shape of each object to be registered; 

a reflectance measurement means for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered; 

25 an image generation means for generating images of each object 

to be registered under various illumination conditions as illumination 
variation images; 

an illumination variation space generation means for generating 
basis vectors of a space that almost accommodates image variation due 

30 to the illumination variation by conducting principal component analysis 
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to the illumination variation images generated by the image generation 
means; and 

a data storage means for storing, as the registered data, the 
basis vectors generated by the illumination variation space generation 
5 means, and 

the comparison means includes- 

a photographing means for photographing, as an input image, 
the target object to be compared with the data of registered objects 
registered in the registration means; 

10 an illumination condition estimation means for obtaining 

correlations between the input image obtained by the photographing 
means and each of the basis vectors stored in the data storage means, 
generating, as the reference image, the image that is the most similar to 
the input image by use of the basis vectors and based on the correlations; 

15 an image comparison means for comparing each reference image 

generated by the illumination condition estimation means with the input 
image obtained by the photographing means and thereby calculating an 
evaluation value concerning the similarity between the two images; and 
a comparison judgment means for judging whether or not each 

20 of the registered objects registered in the registration means is the object 
photographed by the photographing means based on the evaluation value 
calculated by the image comparison means. 
[0046] 

In accordance with a twenty-third aspect of the present 
25 invention, in one of the second to twenty-second aspects, the 
three-dimensional shape measurement means measures the 
three-dimensional shapes by reading data or drawings. 
[0047] 

In accordance with a twenty-fourth aspect of the present 
30 invention, in one of the second to twenty-third aspects, the reflectance 
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measurement means obtains the reflectance by reading data or 
drawings. 
[0048] 

In accordance with a twenty-fifth aspect of the present invention, 
5 in one of the second to twenty-fourth aspects, the photographing 
means obtains the input image by scanning a film, a photograph or 
printed matter. 
[0049] 

In accordance with a twenty-sixth aspect of the present 
10 invention, in one of the second to twenty-fifth aspects, the comparison 
judgment means judges which registered object matches the target 
object. 
[0050] 

In accordance with a twenty-seventh aspect of the present 
15 invention, in one of second to twenty-sixth aspects, the comparison 
judgment means searches for one or more registered objects that are 
similar to the target object. 
[0051] 

In accordance with a twenty-eighth aspect of the present 
20 invention, in one of the first to twenty-seventh aspects, the registered 
objects are automobiles. 
[0052] 

In accordance with a twenty-ninth aspect of the present 
invention, in one of the first to twenty-eighth aspects, the objects to be 
25 registered are human faces. 
[0053] 

In accordance with a thirtieth aspect of the present invention, 
there is provided the steps of 

a registration step in which three-dimensional data of each 
30 object to be registered are registered; and 
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a comparison step in which two-dimensional data of an object to 
be compared with the data registered in the registration step is obtained, 
and the obtained data is compared with the data registered in the 
registration step. 
5 [0054] 

In accordance with a thirty-first aspect of the present invention, 
in the thirtieth aspect, the comparison step includes: 

a photographing step in which the comparison target object is 
photographed and thereby an input image is obtained; 
10 a position/pose correction step in which the position/pose of the 

target object obtained in the photographing step is corrected; 

an illumination correction step in which an image of each 
registered object in the same position/pose as the target object in the 
input image and under an illumination condition most similar to that of 
15 the input image is obtained as a reference image by use of the 
position/pose corrected in the position/pose correction step and the data 
registered in the registration step; 

an image comparison step in which each reference image 
generated in the image correction step is compared with the input image 
20 obtained in the photographing step and thereby an evaluation value 
concerning the similarity between the two images is calculated; and 

a comparison judgment step in which it is judged whether or not 
each registered object registered in the registration step is the object 
photographed in the photographing step based on the evaluation value 
25 calculated in the image comparison step. 
[0055] 

In accordance with a thirty-second aspect of the present 
invention, in the thirty-first aspect, the registration step includes : 

a three-dimensional shape measurement step in which the 
30 three-dimensional shape of each object to be registered is measured; 
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a reflectance measurement step in which the surface reflectance 
at each position of the three-dimensional shape of the object to be 
registered is measured; and 

a data storage step in which the three-dimensional shapes 
5 measured in the three-dimensional shape measurement step and the 
reflectance measured in the reflectance measurement step are stored as 
the registered data. 
[0056] 

In accordance with a thirty-third aspect of the present invention, 
10 in the thirty-first aspect, the registration step includes: 

a three-dimensional shape measurement step in which the 
three-dimensional shape of each object to be registered is measured; 

a image information obtaining step in which the object to be 
registered is photographed and thereby image information of the object 
15 to be registered is obtained; and 

a data storage step in which the three-dimensional shape 
measured in the three-dimensional shape measurement step and the 
image information obtained in the image obtaining step are stored as the 
registered data. 
20 [0057] 

In accordance with a thirty-fourth aspect of the present 
invention, in the thirty-first aspect, the registration step includes: 

a three-dimensional shape measurement step in which the 
three-dimensional shape of each object to be registered is measured; 
25 an average shape generation step in which an average 

three-dimensional shape is generated as the average of a plurality of the 
three-dimensional shapes measured in the three-dimensional shape 
measurement step when registering a plurality of objects; 

a reflectance measurement step in which the surface reflectance 
30 at each position of the three-dimensional shape of the object to be 
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registered is measured; and 

a data storage step in which the three-dimensional shapes 
measured in the three-dimensional shape measurement step, the 
average three-dimensional shape generated in the average shape 
5 generation step and the reflectance measured in the reflectance 
measurement step are stored as the registered data. 
[00058] 

In accordance with a thirty-fifth aspect of the present invention, 
in one of the thirty-second to thirty-fourth aspects, the illumination 
10 correction step includes: 

an image generation step in which images of each registered 
object in the same position/pose as the target object photographed by the 
photographing means and under various illumination conditions are 
generated as illumination variation images by use of the position/pose 
15 corrected in the position/pose correction step and the data registered in 
the registration step! and 

an illumination condition estimation step in which an image 
that is the most similar to the input image obtained in the 
photographing step is generated as the reference image by use of the 
20 illumination variation images generated in the image generation step. 
[0059] 

In accordance with a thirty-sixth aspect of the present invention, 
in the thirty -fifth aspect, 

the illumination correction step further includes an illumination 
25 variation space generation step in which an illumination variation space 
which is spanned by the illumination variation images generated in the 
image generation step is generated, and 

in the illumination condition estimation step, the image that is 
the most similar to the input image is generated as the reference image 
30 from the illumination variation space generated in the illumination 
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variation space generation step. 
[0060] 

In accordance with a thirty-seventh aspect of the present 
invention, in the thirty-sixth aspect, 
5 in the illumination variation space generation step, basis 

vectors of a space that almost accommodates image variation due to the 
illumination variation are generated by conducting principal component 
analysis to the illumination variation images generated in the image 
generation step, and 
10 in the illumination condition estimation step, correlations are 

obtained between the input image obtained in the photographing step 
and each of the basis vectors generated in the illumination variation 
space generation step, and the image that is the most similar to the 
input image is generated as the reference image by use of the basis 
15 vectors and based on the correlations. 
[0061] 

In accordance with a thirty-eighth aspect of the present 
invention, in one of the thirty-fifth to thirty-seventh aspects, the 
illumination correction step further includes an illumination condition 
20 variation step in which various illumination conditions are set and 
generated to be used for generating illumination variation images before 
the illumination variation images are generated in the image generation 
step. 

[0062] 

25 In accordance with a thirty-ninth aspect of the present invention, 

in the thirty-first aspect, 

the registration step includes^ 

a , three-dimensional shape measurement step in which the 
three-dimensional shape of each object to be registered is measured; 
30 a texture image photographing step in which the object to be 



•■ 61 

registered is photographed under various illumination conditions and 
thereby texture images are obtained; and 

a data storage step in which the three-dimensional shapes 
measured in the three-dimensional shape measurement step and the 
5 texture images obtained in the texture image photographing step are 
stored as the registered data, and 

the illumination correction step includes : 

an image generation step in which illumination variation 
images of each registered object in the same position/pose as the target 
10 object in the input image and under various illumination conditions are 
generated by use of the position/pose corrected in the position/pose 
correction step and the three-dimensional shapes and the texture images 
of the object stored in the storage step; and 

an illumination condition estimation step in which an image 
15 that is the most similar to the input image obtained in the 
photographing step is generated as the reference image by use of the 
illumination variation images generated in the image generation step. 
[0063] 

In accordance with a fortieth aspect of the present invention, in 
20 the thirty-ninth aspect, the illumination correction step further includes 
an illumination variation space generation step in which an illumination 
variation space which is spanned by the illumination variation images 
generated in the image generation step is generated before the reference 
image is generated in the illumination condition estimation step, and 
25 in the illumination condition estimation step, the image that is 

the most similar to the input image is generated as the reference image 
from the illumination variation space generated in the illumination 
variation space generation step. 
[0064] 

30 In accordance with a forty-first aspect of the present invention, 
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in the fortieth aspect, 

in the illumination variation space generation step, basis 
vectors of a space that almost accommodates image variation due to the 
illumination variation are generated by conducting the principal 
5 component analysis to the illumination variation images generated in 
the image generation step, and 

in the illumination condition estimation step, correlations are 
obtained between the input image obtained in the photographing step 
and each of the basis vectors generated in the illumination variation 
10 space generation step, and the image that is the most similar to the 
input image is generated as the reference image by use of the basis 
vectors and based on the correlations. 
[0065] 

In accordance with a forty-second aspect of the present 
15 invention, in the thirty-first aspect, in the position/pose correction step, 
the position/pose of the object is the corrected position/pose of a 
predetermined position/pose. 
[0066] 

In accordance with a forty-third aspect of the present invention, 
20 in the thirty-first aspect, in the position/pose correction step, the 
position/pose is a corrected position/pose to which position/pose 
parameters of the object photographed by the photographing means are 
input. 
[0067] 

25 In accordance with a forty-fourth aspect of the present invention, 

in the thirty-first aspect, in the position/pose correction step, the 
position/pose is a corrected position/pose of the estimated position/pose of 
the object photographed by the photographing means. 
[0068] 

30 In accordance with a forty-fifth aspect of the present invention, 
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in the forty-fourth aspect, 

the registration step includes^ 

a three-dimensional shape measurement step in which the 
three-dimensional shape of each object to be registered is measured; 
5 a reflectance measurement step in which the surface reflectance 

at each position of the three-dimensional shape of the object to be 
registered is measured; 

a first feature point extraction step in which feature point 
positions of the registered object are extracted based on the 
10 three-dimensional shape measured in the three-dimensional shape 
measurement step and the reflectance measured in the reflectance 
measurement step; and 

a data storage step in which the three-dimensional shapes 
measured in the three-dimensional shape measurement step, the 
15 reflectance measured in the reflectance measurement step and the 
feature point positions extracted in the first feature point extraction step 
are stored as the registered data, and 

the position/pose correction step includes : 

a second feature point extraction step in which feature point 
20 positions same as those stored in the data storage step are extracted 
from the input image obtained in the photographing step as input image 
feature point positions; and 

a position/pose calculation step in which the position/pose of the 
object photographed in the photographing means is estimated based on 
25 the three-dimensional shape and the feature point positions stored in the 
data storage step and the input image feature point positions obtained in 
the second feature point extraction step. 
[0069] 

In accordance with a forty-sixth aspect of the present invention, 
30 in the forty-fifth aspect, the illumination correction step includes^ 
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an image generation step in which images of each object in the 
same position/pose as the object obtained in the photographing step and 
under various illumination conditions are generated as illumination 
variation images by use of the position/pose estimated in the 
5 position/pose determination step and the three-dimensional shape and 
the reflectance stored in the data storage step; and 

an illumination condition estimation step in which an image 
that is the most similar to the input image obtained in the 
photographing step is generated as the reference image by use of the 
10 illumination variation images generated in the image generation step. 
[0070] 

In accordance with a forty-seventh aspect of the present 
invention, in the forty-sixth aspect, the illumination correction step 
further includes an illumination variation space generation step in which 
15 an illumination variation space which is spanned by the illumination 
variation images generated in the image generation step is generated 
before the reference image is generated in the illumination condition 
estimation step, and 

in the illumination condition estimation step, the image that is 
20 the most similar to the input image is generated as the reference image 
from the illumination variation space generated in the illumination 
variation space generation step. 
[0071] 

In accordance with a forty-eighth aspect of the present invention, 
25 in the forty-seventh aspect, 

in the illumination variation space generation step, basis 
vectors of a space that almost accommodates image variation due to the 
illumination variation are generated by conducting principal component 
analysis to the illumination variation images generated in the image 
30 generation step, and 
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in the illumination condition estimation step, correlations are 
obtained between the input image obtained in the photographing step 
and each of the basis vectors generated in the illumination variation 
space generation step, and the image that is the most similar to the 
5 input image obtained in the photographing step is generated as the 
reference image by use of the basis vectors and based on the correlations. 
[0072] 

In accordance with a forty-ninth aspect of the present invention, 
in one of the forty-sixth to forty-eighth aspects, the illumination 
10 correction step further includes an illumination condition variation step 
in which various illumination conditions are set and generated before 
illumination variation images are generated in the image generation 
step. 

[0073] 

15 In accordance with a fiftieth aspect of the present invention, in 

the thirtieth aspect, 

the registration step includes- 

a three-dimensional shape measurement step in which the 
three-dimensional shape of each object to be registered is measured; 
20 a reflectance measurement step in which the surface reflectance 

at each position of the three-dimensional shape of the object to be 
registered is measured; 

an image generation step in which images of each object to be 
registered under various illumination conditions are generated as 
25 illumination variation images; 

an illumination variation space generation step in which an 
illumination variation space which is spanned by the illumination 
variation images generated in the image generation step is generated; 
and 

30 a data storage step in which the illumination variation space 
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generated in the illumination variation space generation step is stored as 
the registered data, and 

the comparison step includes: 

a photographing step in which the comparison target object to be 
5 compared with the registered object registered in the registration step is 
photographed as an input image is obtained; 

an illumination condition estimation step in which an image of 
each object that is the most similar to the input image obtained in the 
photographing step is generated as a reference image from the 
10 illumination variation space stored in the data storage step; 

an image comparison step in which each reference image 
generated in the illumination condition estimation step is compared with 
the input image obtained in the photographing step and thereby an 
evaluation value concerning the similarity between the two images is 
15 calculated; and 

a comparison judgment step in which it is judged whether or not 
each of the registered objects registered in the registration step is the 
target object photographed in the photographing step based on the 
evaluation value calculated in the image comparison step. 
20 [0074] 

In accordance with a fifty-first aspect of the present invention, 
in the thirtieth aspect, 

the registration step includes^ 

a three-dimensional shape measurement step for measuring the 
25 three-dimensional shape of each object to be registered; 

a reflectance measurement step for measuring the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered; 

an image generation step for generating images of each object to 
30 be registered under various illumination conditions as illumination 
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variation images; 

an illumination variation space generation step for generating 
basis vectors of a space that almost accommodates image variation due 
to the illumination variation by conducting principal component analysis 
5 to the illumination variation images generated in the image generation 
step; and 

a data storage step for storing, as the registered data, the basis 
vectors generated by the illumination variation space generation step, 
and 

10 the comparison step includes : 

a photographing step for photographing, as an input image, the 
target object to be compared with the data of registered objects 
registered in the registration step; 

an illumination condition estimation step for obtaining 
15 correlations between the input image obtained by the photographing step 
and each of the basis vectors stored in the data storage step, generating, 
as the reference image, the image that is the most similar to the input 
image by use of the basis vectors and based on the correlations; 

an image comparison step for comparing each reference image 
20 generated by the illumination condition estimation step with the input 
image obtained by the photographing step and thereby calculating an 
evaluation value concerning the similarity between the two images; and 

a comparison judgment step for judging whether or not each of 
the registered objects registered in the registration step is the object 
25 photographed by the photographing step based on the evaluation value 
calculated by the image comparison step. 
[0075] 

In accordance with a fifty-second aspect of the present invention, 
in one of the thirty-first to fifty-first aspects, the three-dimensional 
30 shape measurement step measures the three-dimensional shapes by 
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reading data or drawings. 
[0076] 

In accordance with a fifty-third aspect of the present invention, 
in one of the thirty-first to fifty-second aspects, the reflectance 
5 measurement step obtains the reflectance by reading data or drawings. 
[0077] 

In accordance with a fifty-fourth aspect of the present invention, 
in one of the thirty-first to fifty-third aspects, the photographing step 
obtains the input image by scanning a film, a photograph or printed 
10 matter. 

[0078] 

In accordance with a fifty-fifth aspect of the present invention, 
in one of the thirty-first to fifty-fourth aspects, the comparison judgment 
step judges which registered object matches the target object. 
15 [0079] 

In accordance with a fifth-sixth aspect of the present invention, 
in one of the thirty-first to fifty-fifth aspects, the comparison judgment 
step searches for one or more registered objects that are similar to the 
target object. 
20 [0080] 

In accordance with a fifty-seventh aspect of the present 
invention, in one of the thirtieth to fifty-sixth aspects, the registered 
objects are automobiles. 
[0081] 

25 In accordance with a fifty-eighth aspect of the present invention, 

in one of the thirtieth to fifty-seventh aspects, the objects to be registered 
are human faces. 
[0082] 

In accordance with a fifty-ninth aspect of the present invention, 
30 there is executing the processes of a registration process in which 
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three-dimensional data of each object to be registered are registered; and 
a comparison process in which two-dimensional data of an object 
to be compared with the data registered in the registration step is 
obtained, and the obtained data is compared with the data registered in 
5 the registration process. 
[0083] 

In accordance with a sixtieth aspect of the present invention, in 
the fifty-ninth aspect, the comparison process executes the processes of: 

a photographing process in which the comparison target object 
10 is photographed and thereby an input image is obtained; 

a position/pose correction process in which the position/pose of 
the target object obtained in the photographing process is corrected; 

an illumination correction process in which an image of each 
registered object in the same position/pose as the target object in the 
15 input image and under an illumination condition most similar to that of 
the input image is obtained as a reference image by use of the 
position/pose corrected in the position/pose correction process and the 
data registered in the registration process; 

an image comparison process in which each reference image 
20 generated in the image correction process is compared with the input 
image obtained in the photographing process and thereby an evaluation 
value concerning the similarity between the two images is calculated; 
and 

a comparison judgment process in which it is judged whether or 
25 not each registered object registered in the registration process is the 
object photographed in the photographing process based on the 
evaluation value calculated in the image comparison process. 
[0084] 

In accordance with a sixty-first aspect of the present invention, 
30 in the sixtieth aspect, the registration process executes the processes of 
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a three-dimensional shape measurement process in which the 
three-dimensional shape of each object to be registered is measured; 

a reflectance measurement process in which the surface 
reflectance at each position of the three-dimensional shape of the object 
5 to be registered is measured; and 

a data storage process in which the three-dimensional shapes 
measured in the three-dimensional shape measurement process and the 
reflectance measured in the reflectance measurement process are stored 
as the registered data. 
10 [0085] 

In accordance with a sixty-second aspect of the present 
invention, in the sixtieth aspect, the registration process executes the 
processes of 

a three-dimensional shape measurement process in which the 
15 three-dimensional shape of each object to be registered is measured; 

a image information obtaining process in which the object to be 
registered is photographed and thereby image information of the object 
to be registered is obtained; and 

a data storage process in which the three-dimensional shape 
20 measured in the three-dimensional shape measurement process and the 
image information obtained in the image obtaining process are stored as 
the registered data. 
[0086] 

In accordance with a sixty-third aspect of the present invention, 
25 in the sixtieth aspect, the registration process executes the processes of 
a three-dimensional shape measurement process in which the 
three-dimensional shape of each object to be registered is measured; 

an average shape generation process in which an average 
three-dimensional shape is generated as the average of a plurality of the 
30 three-dimensional shapes measured in the three-dimensional shape 
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measurement process when registering a plurality of objects! 

a reflectance measurement process in which the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered is measured; and 
5 a data storage process in which the three-dimensional shapes 

measured in the three-dimensional shape measurement process, the 
average three-dimensional shape generated in the average shape 
generation process and the reflectance measured in the reflectance 
measurement process are stored as the registered data. 
10 [0087] 

In accordance with a sixty-fourth aspect of the present invention, 
in one of the sixty-first to sixty-third aspects, the illumination correction 
process executes the processes of 

an image generation process in which images of each registered 
15 object in the same position/pose as the target object photographed by the 
photographing means and under various illumination conditions are 
generated as illumination variation images by use of the position/pose 
corrected in the position/pose correction process and the data registered 
in the registration process; and 
20 an illumination condition estimation process in which an image 

that is the most similar to the input image obtained in the 
photographing process is generated as the reference image by use of the 
illumination variation images generated in the image generation process. 
[0088] 

25 In accordance with a sixty-fifth aspect of the present invention, 

in the sixty-fourth aspect, the illumination correction process further 
includes an illumination variation space generation process in which an 
illumination variation space which is spanned by the illumination 
variation images generated in the image generation process is generated, 

30 and 
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in the illumination condition estimation process, the image that 
is the most similar to the input image is generated as the reference 
image from the illumination variation space generated in the 
illumination variation space generation process. 
5 [0089] 

In accordance with a sixty-sixth aspect of the present invention, 
in the sixty-fifth aspect, in the illumination variation space generation 
process, basis vectors of a space that almost accommodates image 
variation due to the illumination variation are generated by conducting 
10 principal component analysis to the illumination variation images 
generated in the image generation process, and 

in the illumination condition estimation process, correlations 
are obtained between the input image obtained in the photographing 
process and each of the basis vectors generated in the illumination 
15 variation space generation process, and the image that is the most 
similar to the input image is generated as the reference image by use of 
the basis vectors and based on the correlations. 
[0090] 

In accordance with a sixty-seventh aspect of the present 
20 invention, in one of the sixty-fourth to sixty-sixth aspects, the 
illumination correction process further includes an illumination 
condition variation process in which various illumination conditions are 
set and generated to be used for generating illumination variation 
images before the illumination variation images are generated in the 
25 image generation process. 
[0091] 

In accordance with a sixty-eighth aspect of the present invention, 
in the sixtieth aspect, the registration process includes: 

a three-dimensional shape measurement process in which the 
30 three-dimensional shape of each object to be registered is measured; 
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a texture image photographing process in which the object to be 
registered is photographed under various illumination conditions and 
thereby texture images are obtained; and 

a data storage process in which the three-dimensional shapes 
5 measured in the three-dimensional shape measurement process and the 
texture images obtained in the texture image photographing process are 
stored as the registered data, and 

the illumination correction process includes-* 

an image generation process in which illumination variation 
10 images of each registered object in the same position/pose as the target 
object in the input image and under various illumination conditions are 
generated by use of the position/pose corrected in the position/pose 
correction process and the three-dimensional shapes and the texture 
images of the object stored in the storage process; and 
15 an illumination condition estimation process in which an image 

that is the most similar to the input image obtained in the 
photographing process is generated as the reference image by use of the 
illumination variation images generated in the image generation process. 
[0092] 

20 In accordance with a sixty-ninth aspect of the present invention, 

in the sixty-eighth aspect, the illumination correction process further 
includes an illumination variation space generation process in which an 
illumination variation space which is spanned by the illumination 
variation images generated in the image generation process is generated 

25 before the reference image is generated in the illumination condition 
estimation process, and 

in the illumination condition estimation process, the image that 
is the most similar to the input image is generated as the reference 
image from the illumination variation space generated in the 

30 illumination variation space generation process. 
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[0093] 

In accordance with a seventieth aspect of the present invention, 
in the sixty-ninth aspect, in the illumination variation space generation 
process, basis vectors of a space that almost accommodates image 
variation due to the illumination variation are generated by conducting 
the principal component analysis to the illumination variation images 
generated in the image generation process, and 

in the illumination condition estimation process, correlations 
are obtained between the input image obtained in the photographing 
process and each of the basis vectors generated in the illumination 
variation space generation process, and the image that is the most 
similar to the input image is generated as the reference image by use of 
the basis vectors and based on the correlations. 
[0094] 

In accordance with a seventy-first aspect of the present 
invention, in the sixtieth aspect, in the position/pose correction process, 
the position/pose of the object is the corrected position/pose of a 
predetermined position/pose. 
[0095] 

In accordance with a seventy-second aspect of the present 
invention, in the sixtieth aspect, in the position/pose correction process, 
the position/pose is a corrected position/pose to which position/pose 
parameters of the object photographed by the photographing means are 
input. 
[0096] 

In accordance with a seventy-third aspect of the present 
invention, in the sixtieth aspect, in the position/pose correction process, 
the position/pose is a corrected position/pose of the estimated 
position/pose of the object photographed by the photographing means. 
[0097] 
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In accordance with a seventy-fourth aspect of the present 
invention, in seventy-third aspect, 

the registration process includes* 

a three-dimensional shape measurement process in which the 
5 three-dimensional shape of each object to be registered is measured; 

a reflectance measurement process in which the surface 
reflectance at each position of the three-dimensional shape of the object 
to be registered is measured; 

a first feature point extraction process in which feature point 
10 positions of the registered object are extracted based on the 
three-dimensional shape measured in the three-dimensional shape 
measurement process and the reflectance measured in the reflectance 
measurement process; and 

a data storage process in which the three-dimensional shapes 
15 measured in the three-dimensional shape measurement process, the 
reflectance measured in the reflectance measurement process and the 
feature point positions extracted in the first feature point extraction 
process are stored as the registered data, and 

the position/pose correction process includes- 
20 a second feature point extraction process in which feature point 

positions same as those stored in the data storage process are extracted 
from the input image obtained in the photographing process as input 
image feature point positions; and 

a position/pose calculation process in which the position/pose of 
25 the object photographed in the photographing means is estimated based 
on the three-dimensional shape and the feature point positions stored in 
the data storage process and the input image feature point positions 
obtained in the second feature point extraction process. 
[0098] 

30 In accordance with a seventy-fifth aspect of the present 
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invention, in the seventy-fourth aspect, the illumination correction 
process includes: 

an image generation process in which images of each object in 
the same position/pose as the object obtained in the photographing 
5 process and under various illumination conditions are generated as 
illumination variation images by use of the position/pose estimated in 
the position/pose determination process and the three-dimensional shape 
and the reflectance stored in the data storage process; and 

an illumination condition estimation process in which an image 
10 that is the most similar to the input image obtained in the 
photographing process is generated as the reference image by use of the 
illumination variation images generated in the image generation process. 
[0099] 

In accordance with a seventy-sixth aspect of the present 
15 invention, in the seventy-fifth aspect, the illumination correction process 
further includes an illumination variation space generation process in 
which an illumination variation space which is spanned by the 
illumination variation images generated in the image generation process 
is generated before the reference image is generated in the illumination 
20 condition estimation process, and 

in the illumination condition estimation process, the image that 
is the most similar to the input image is generated as the reference 
image from the illumination variation space generated in the 
illumination variation space generation process. 
25 [0100] 

In accordance with a seventy-seventh aspect of the present 
invention, in the seventy-sixth aspect, in the illumination variation 
space generation process, basis vectors of a space that almost 
accommodates image variation due to the illumination variation are 
30 generated by conducting principal component analysis to the 
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illumination variation images generated in the image generation process, 
and 

in the illumination condition estimation process, correlations 
are obtained between the input image obtained in the photographing 
5 process and each of the basis vectors generated in the illumination 
variation space generation process, and the image that is the most 
similar to the input image obtained in the photographing process is 
generated as the reference image by use of the basis vectors and based 
on the correlations. 
10 [0101] 

In accordance with a seventy-eighth aspect of the present 
invention, in one of the seventy-fifth to seventy-seventh aspects, the 
illumination correction process further includes an illumination 
condition variation process in which various illumination conditions are 
15 set and generated before illumination variation images are generated in 
the image generation process. 
[0102] 

In accordance with a seventy-ninth aspect of the present 
invention, in the fifty-ninth aspect, 
20 the registration process includes- 

a three-dimensional shape measurement process in which the 
three-dimensional shape of each object to be registered is measured; 

a reflectance measurement process in which the surface 
reflectance at each position of the three-dimensional shape of the object 
25 to be registered is measured; 

an image generation process in which images of each object to be 
registered under various illumination conditions are generated as 
illumination variation images; 

an illumination variation space generation process in which an 
30 illumination variation space which is spanned by the illumination 
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variation images generated in the image generation process is generated; 
and 

a data storage process in which the illumination variation space 
generated in the illumination variation space generation process is 
5 stored as the registered data, and 

the comparison process includes: 

a photographing process in which the comparison target object 
to be compared with the registered object registered in the registration 
process is photographed as an input image is obtained; 

10 an illumination condition estimation process in which an image 

of each object that is the most similar to the input image obtained in the 
photographing process is generated as a reference image from the 
illumination variation space stored in the data storage process; 

an image comparison process in which each reference image 

15 generated in the illumination condition estimation process is compared 
with the input image obtained in the photographing process and thereby 
an evaluation value concerning the similarity between the two images is 
calculated; and 

a comparison judgment process in which it is judged whether or 
20 not each of the registered objects registered in the registration process is 
the target object photographed in the photographing process based on 
the evaluation value calculated in the image comparison process. 
[0103] 

In accordance with a eightieth aspect of the present invention, 
25 in the fifty- ninth aspect, 

the registration process includes^ 

a three-dimensional shape measurement process for measuring 
the three-dimensional shape of each object to be registered; 

a reflectance measurement process for measuring the surface 
30 reflectance at each position of the three-dimensional shape of the object 
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to be registered; 

an image generation process for generating images of each 
object to be registered under various illumination conditions as 
illumination variation images; 
5 an illumination variation space generation process for 

generating basis vectors of a space that almost accommodates image 
variation due to the illumination variation by conducting principal 
component analysis to the illumination variation images generated in 
the image generation process; and 
10 a data storage process for storing, as the registered data, the 

basis vectors generated by the illumination variation space generation 
process, and 

the comparison process includes • 

a photographing process for photographing, as an input image, 
15 the target object to be compared with the data of registered objects 
registered in the registration process; 

an illumination condition estimation process for obtaining 
correlations between the input image obtained by the photographing 
process and each of the basis vectors stored in the data storage process, 
20 generating, as the reference image, the image that is the most similar to 
the input image by use of the basis vectors and based on the correlations; 

an image comparison process for comparing each reference 
image generated by the illumination condition estimation process with 
the input image obtained by the photographing process and thereby 
25 calculating an evaluation value concerning the similarity between the 
two images; and 

a comparison judgment process for judging whether or not each 
of the registered objects registered in the registration process is the 
object photographed by the photographing process based on the 
30 evaluation value calculated by the image comparison process. 



80 

[0104] 

In accordance with an eighty-first aspect of the present 
invention, in one of the sixtieth to eightieth aspects, the 
three-dimensional shape measurement process measures the 
5 three-dimensional shapes by reading data or drawings. 
[0105] 

In accordance with an eighty-second aspect of the present 
invention, in one of the sixtieth to eighty-first aspects, the reflectance 
measurement process obtains the reflectance by reading data or 
10 drawings. 

[0106] 

In accordance with an eighty-third aspect of the present 
invention, in one of the sixtieth to eighty-second aspects, the 
photographing process obtains the input image by scanning a film, a 
15 photograph or printed matter. 
[0107] 

In accordance with an eighty-fourth aspect of the present 
invention, in one of the sixtieth to eighty-third aspects, the comparison 
judgment process judges which registered object matches the target 
20 object. 

[0108] 

In accordance with an eighty-fifth aspect of the present 
invention, in one of the sixtieth to eighty-fourth aspects, the comparison 
judgment process searches for one or more registered objects that are 
25 similar to the target object. 
[0109] 

In accordance with an eighty-sixth aspect of the present 
invention, in one of the fifty- ninth to eighty-fifth aspects, the objects to 
be registered are automobiles. 
30 [0110] 
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In accordance with an eighty-seventh aspect of the present 
invention, in one of fifty-nine to eighty-sixth aspects, the objects to be 
registered are human faces. 
[0111] 

5 [Embodiments of the Invention] 

Referring now to the drawings, a description will be given in 
detail of preferred embodiments in accordance with the present 
invention. 
[0112] 

10 (First Embodiment) 

Fig. 1 is a block diagram showing the composition and procedure 
of an image comparison device in accordance with a first embodiment of 
the present invention. Fig. 2 is a composition diagram of this 
embodiment. As shown in Fig. 1, a registration means 100 measures 

15 the three-dimensional shape of each object and the reflectance or color on 
the surface of the object by use of a three-dimensional shape 
measurement device, and stores both of the data. A comparison means 
200 photographs a two-dimensional image by use of a video camera, etc., 
and compares the two-dimensional image with the data stored in the 

20 registration means 100. 
[0113] 

The image comparison device of the first embodiment employs 
the following principles that an image of an object in an arbitrary 
position/pose under an arbitrary illumination condition can be generated 
25 by preliminarily registering the three-dimensional shape and surface 
reflectance of the object, and a set of images of the object in the 
position/pose under various illumination conditions can be expressed as a 
low-dimensional subspace of the image space. 
[0114] i 

30 The registration means 100 includes a three-dimensional shape 
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measurement means 110, a reflectance measurement means 120 and a 
data storage means 130. 
[0115] 

The three-dimensional shape measurement means 110 measures 
5 the three-dimensional shape of each object and outputs the 
three-dimensional shape to the data storage means 130. For example, a 
three-dimensional shape measurement device which has been disclosed 
in Japanese Patent Application No.HEI 11- 123687 can be employed. 
Other three-dimensional shape measurement devices can be employed. 
10 [0116] 

The reflectance measurement means 120 measures the 
reflectance of the surface of the object corresponding to the 
three-dimensional shape, and outputs the reflectance data to the data 
storage means 130. For, example, if the three-dimensional shape 

15 measurement device disclosed in Japanese Patent Application 
No.HEI 11- 123687 is employed, color information on the surface of the 
object can be measured simultaneously with the three-dimensional shape. 
Hereinafter, the color information is substituted for the reflectance data. 
For example, there is employed the fact that an image being 

20 photographed under such illumination conditions where the object is 
illuminated uniformly and any shadow is hardly cast, its intensity value 
is considered to be substantially in proportion to its reflectance. 
[0117] 

As a concrete method, a hemispherical scaffold is set up in front 
25 of the object to be registered and an appropriate number of illumination 
lamps are fixed to the scaffold, and an image is photographed with all 
lamps being turned on at the same time. Other than this method, there 
are various methods as to employing one or more reflectors in order to 
diffuse light and illuminate the object uniformly. 
30 [0118] 
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The data storage means 130 stores the three-dimensional shape 
data and the reflectance data of each registered object. The data stored 
in the data storage means 130 are read out at a proper timing for the 
comparison process by the comparison means 200. 
5 [0119] 

The comparison means 200 includes a photographing means 210, 
a position/pose estimation means 220, an illumination correction means 
230, an image comparison means 240 and a judgment means 250. 
[0120] 

10 The photographing means 210 which is provided with a camera, 

a video camera, etc. photographs a target object for correction and 
outputs the obtained image to the position/pose estimation means 220, 
the illumination correction means 230 and the image comparison means 
240 as an input image. 

15 [0121] 

The position/pose estimation means 220 estimates 
photographing conditions such as parameters of position/pose of the 
object and photographing device at the time when the input image was 
photographed. For example, the position/pose parameters include the 

20 translation distance (Tx, Ty, Tz) of the object, rotation angles (Rx, Ry, Rz), 
the focal length "f 1 of the camera, and the viewing angle " a ". The 
position/pose estimation means 220 is provided with an interactive 
interface by which the user adjusts the position/pose parameters 
manually watching a display. 

25 [0122] 

For example, an image of a comparison target object generated 
by computer graphics by use of the above 8 position/pose parameters and 
the input image are displayed by being superimposed on the screen. 
The user successively adjusts the parameters so that the two images will 
30 be in register and thereby appropriate parameters are determined. The 
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interactive interface is only an example and various types of devices can 
be employed as the position/pose estimation means 220. It is also 
possible to let the position/pose estimation means 220 calculate the 
position/pose parameters automatically. 
5 [0123] 

In the case of the automatic parameter estimation, the 
position/pose estimation means 220 successively generates CG images of 
the comparison target object in various positions/poses, compares the CG 
images with the input image, selects a CG image that matches the input 
10 image the best, and thereby obtains the parameters of position/pose and 
the photographing device, for example. 
[0124] 

The illumination correction means 230 generates, by use of the 
above parameters determined by the position/pose estimation means 220, 
15 an image, as a reference image, which is in the same position/pose as 
that of the input image and is under an illumination condition that is the 
most similar to that of the input image. In the following, the 
illumination correction process will be explained in detail. 
[0125] 

20 Assuming that the reflectance property on the surface of the 

object is perfect scattering and the surface is convex with no shadow 
caused by light occlusion by other parts and the light source exists at an 
infinite-point, the intensity value I(u, v) of each pixel (u, v) of an image 
can be expressed as the following [Equation l], by use of the reflectance 

25 R(u, v) and a normal vector N(u, v) corresponding to the pixel (u, v) and 
the intensity Ii and a direction vector Li: 

[0126] 
[Equation l] 



/(//, v) = B(u t v)max^('i * * ")).<> 



(1) 
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[0127] 

Here, if we neglect the effect of the "max 0", an arbitrary 
illumination condition including cases where a plurality of lamps are 
used can be expressed by only one illumination vector L as follows- 

[0128] 
[Equation 2] 



l(u,v) = B(u,v)N{u,v). lit = 



*(2) 



[0129] 

The degree of freedom of images of the object generated by the 
10 illumination variation is equal to the number of dimension of the 
illumination vector L, that is, 3 or less. However, the degree of freedom 
actually becomes higher due to the effects of the "max 0", cast shadow 
occurring on the surface of the object due to light occlusion by other parts, 
imperfect scattering on the surface, etc. However, most of the images 
15 lie in or lie sufficiently close to a three-dimensional subspace. Therefore, 
actually occurring image variation can be approximated as a 
low-dimensional subspace. The low-dimensional subspace will be 
hereinafter referred to as an "illumination variation space" of each 
object. 
20 [0130] 

The principal component analysis is used for obtaining basis 
vectors of the illumination variation space. A plurality of images of the 
target object generated by the illumination variation (hereinafter, 
referred to as "illumination variation images") are prepared, and a set 
25 being composed of all the images generated by the variation of 
illumination condition is approximated. Each of the illumination 
variation images is generated as an image under illumination of a single 
point source of light at an infinite-point. A plurality of illumination 
variation images are prepared by setting and changing the direction of 
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the point source of light at appropriate intervals so that all possible 
directions of the point sources in the photographing of the input image 
can be covered. An image obtained under plurality of illumination 
lamps can be expressed as the sum of images obtained under each single 
5 illumination lamp, therefore, the illumination variation images each of 
which is generated under a single point source of light are enough. For 
the generation of the illumination variation images, the 
three-dimensional shape data and the surface reflectance data which 
have been stored in the data storage means 130 are used. 
10 [0131] 

The illumination variation images can be generated by use of 
basic functions of computer graphics, for example. The details of the 
computer graphics functions have been described in a document: Mason 
Woo, Jackie Neider, and Tom Davis "Open GL programming Guide", 

15 Addison-Wesley Publishers Japan. In a standard function provided to 
computers, the reflectance property of the surface of the object is 
generally assumed to be perfect scattering and only shade is generated. 
However, in this embodiment, a reflectance model that is close to actual 
surface reflectance property of the object is selected and used. The 

20 mirror reflection is taken into consideration and cast shadows are also 
reproduced by means of ray tracing, in order to generate images that are 
close to reality. 
[0132] 

The above illumination variation image generation method by 
25 use of the CG functions of a computer is only an example. It is of course 
possible to generate each illumination variation image by calculating 
intensity of pixels that will be used for the comparison, by means of 
numerical calculations. 
[0133] 

30 Hereafter, each image will be expressed by use of a column 
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vector. Each component of the column vector indicates an intensity 
value of each pixel in an area of the image that will be used for the 
comparison. When the number of the illumination variation images is 
N, by expressing each illumination variation image by a vector Ki (i = 1, 
2, • • • , N), a covariance matrix V can be expressed by the following 
[Equation 3]. 

[0134] 
[Equation 3] 



S = 



V = —SS T -"(3) 
N 



10 [0135] 

Eigenvalues a i and eigenvectors Bi (i = 1, 2, • • • M) of V are 
successively obtained for M largest eigenvalues starting from the largest 
eigenvalue, and the illumination variation space of an object "j" is 
approximated by an M- dimensional linear space TPJ whose bases are the 

15 eigenvectors Bi. The dimension number M of the illumination variation 
space may be determined in consideration of precision that is required of 
the illumination correction process. In the case where M eigenvectors 
are used, "cumulative contribution" of the M eigenvalues can be 
calculated by [Equation 4] as follows: 

20 [0136] 

[Equation 4] 

xl00[%] ■••(4) 
[0137] 

The cumulative contribution is an index indicating how 
25 precisely the illumination variation space can express the illumination 
variation images, when error between images is evaluated by use of the 
differences of intensity values therebetween. By setting a threshold 
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value for this index, the dimension number M can be determined 
automatically by finding a number M necessary for surpassing the 
threshold value. 
[0138] 

5 Fig. 3 is a block diagram showing the detailed composition and 

process flow of the illumination correction means 230. The illumination 
correction means 230 includes an illumination condition variation means 
231, an image generation means 232, an illumination variation space 
generation means 233 and an illumination condition estimation means 
10 234. 

[0139] 

The illumination condition variation means 231 sets a sufficient 
number of illumination conditions capable of approximating the 
illumination variation space of the object. For example, considering one 

15 light source at an infinite-point, and the direction of a point source of 
light is expressed by angles ( 0 , <f> ) which indicate a longitude 9 and a 
latitude <f> of a spherical surface of an object such as shown in Fig. 4. 
Each angle (0, 0) is changed from -90° to 90° at intervals of 10° , 
and thereby, 361 types of illumination conditions are set. The type of 

20 light source, the interval or the range of illumination directions is only 
an example, and other settings can of course be employed. 
[0140] 

The image generation means 232 reads the three-dimensional 
shape data and the reflectance data of the object j being a comparison 

25 target from the data storage means 130, and generates the illumination 
variation images by use of the position/pose parameters supplied from 
the position/pose estimation means 220 and the illumination conditions 
supplied from the illumination condition variation means 231 by use of 
CG functions. 

30 [0141] 
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The above process can be implemented by use of basic functions 
of a computer being provided with graphics functions. In the image 
generation by use of computer graphics, various types of surface 
reflectance models and camera models can be employed. For example, 
5 the pinhole camera model may be employed as the camera model and the 
perfect scattering model can be employed as the surface reflectance 
model. It is also possible to give shadows to the surface of the object by 
means of ray tracing, or to give specularity to the surface of the object by 
use of other reflectance models. 
10 [0142] 

In the image comparison process, reflectance property model, a 
light source model and a camera model are made to be close to reality, 
comparison quality can be improved. The image generation process can 
also be conducted by numerical calculations without using computer 
15 graphics. 

[0143] 

The illumination variation space generation means 233 
calculates illumination variation space from the illumination variation 
images by following the [Equation 3], and outputs the obtained basis 

20 vectors to the illumination condition estimation means 234 as the 
illumination variation space (basis vectors) Wj. In this embodiment, M 
basis vectors corresponding to the M largest eigenvalues are obtained 
starting from the largest eigenvalue, and the obtained M basis vectors 
are outputted as the illumination variation space ¥j. In order to 

25 determine the number of the basis vectors M ,for example, as the number 
which surpass 95% of the cumulative contribution being calculated by 
the [Equation 4], in the case where the number of pixels equals to or less 
than 361 being the number of the illumination variation images, the 
number of pixels is N and N eigenvalues are obtained, and a number M 

30 that satisfies the following [Equation 5] is obtained. 
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[0144] 
[Equation 5] 



-i361 



>0.95 ■■(5) 



I" 

[0145] 

5 M can be determined by applying various criterions. 

[0146] 

The illumination condition estimation means 234 generates, as 
a reference image, which is the nearest to the input image and is in the 
illumination variation space fFj according to the following [Equation 6], 
10 and outputs the generated reference image to the image comparison 
means 240. 

[0147] 
[Equation 6] 

lc=^{l r B^Bi "-(6) 
15 [0148] 

The image comparison means 240 calculates an evaluation value 
concerning the similarity between the input image and the generated 
reference image. Various calculation methods can be employed for the 
evaluation. For example, the sum of squares of difference of intensity 
20 value of each pixel of an image which is shown in the following equation 
(7) can be used as the evaluation value. 

[0149] 
[Equation 7] 



2 



•(7) 



25 [0150] 

A technique elaborated on in a document: Shigeru Akamatsu 
"Computer Recognition of Human Faces - A Survey -", The Transactions 
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of the Institute of Electronics, Information and Communication 
Engineers D- H , Vol.J-80-D- II , No. 8, pp.2131-2146 (1997), can also 
employed. 
[0151] 

5 The judgment means 250 judges whether or not the comparison 

target image is the registered image by processing the calculated 
evaluation value as a threshold value. 
[0152] 

Comparison processing of an object may be performed by use of 
10 the distance of illumination variation space between a input image 
vector Iq and a comparison target object as a measure. This distance 
exists in the illumination variation space ^Fj, and is calculable as the 
distance between the input image and an image vector Ic being closest to 
the input image. While various measures can be used as a measure of a 
15 distance, a description will be given of an example in which a square 
error of intensity value is directly used, here. 
[0153] 

In the case where this distant measure is used, the image vector 
Ic being closest to the vector Iq in the space TPj can be generated by the 
20 [Equation 6] 
[0154] 

A distance D (a sum of square of intensity value) between a 
reference image vector Ic and the input image vector Iq can be calculated 
by the [Equation 7]. 
25 [0155] 

The value D is an evaluation value of similarity between the 
input image and the registered data, and based on this evaluation value, 
judgment processes such as whether or not the target image is the 
registered image, to which registered images the input image 
30 corresponds, to which registered image the input image is similar, are 



performed. For example, for simply judging whether or not the target 
image is the registered image, a threshold value D' is predetermined, and 
the target object is judged to be the registered object if D < D\ 
[0156] 

5 When a plurality of objects have been registered, the process 

from the reference image generation by the illumination correction 
means 230 to the evaluation value calculation by the image comparison 
means 240 are repeated for a plural times. In this case, the judgment 
means 250 can judge which registered object is the most similar to the 

10 target object. It is also possible to search for one or more registered 
objects having more than certain evaluation values as objects similar to 
the target object. 
[0157] 
(Second Embodiment) 

15 Next, a detailed description will be given of a second 

embodiment of the present invention with reference to Figs. 6 and 7. 
Fig. 6 is a block diagram showing the composition and operation of an 
image comparison device in accordance with the second embodiment of 
the present invention. This embodiment differs from the first 

20 embodiment in that by photographing images under a plurality of 
illumination conditions, and by use of the images instead of the 
reflectance, illumination variations images are generated, instead of 
measuring the reflectance by use of the reflectance measurement means 
120 in the first embodiment. Further, in the second embodiment, the 

25 illumination condition variation means is not provided. 
[0158] 

In the second embodiment, there is set an adequate number of 
illumination conditions for generating enough sample images for 
generating illumination variation spaces including the variation of the 
30 image of the comparison target object due to the illumination conditions. 
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By photographing image information under the illumination conditions, 
it is possible to generate sample images for generating illumination 
variation space without the need of the reflectance measurement, 
illumination condition setting in the CG image generation, shadow 
5 generation by means of ray tracing, etc. 
[0159] 

As an example of a method which can be applied to a 
photographing process of texture images to be used for the sample 
images, there is the following method. A hemispherical scaffold is set 

10 up in front of the object to be registered and an appropriate number of 
illumination lamps are fixed to the scaffold at even intervals. The 
object is photographed repeatedly by successively turning each lamp on. 
It is also possible to photograph the object repeatedly moving a lamp by a 
manipulator and turning the lamp on repeatedly. 

15 [0160] 

In the second embodiment, the photographed texture images are 
registered and stored together with the three-dimensional shape data as 
the registered data. The comparison step is conducted without 
executing the illumination condition variation process. Instead of the 

20 illumination condition variation process, the stored texture images are 
successively read out and an image of the registered object in the 
position/pose estimated by the position/pose estimation process is 
generated by means of computer graphics so that intensity values on the 
surface of the object in the generated image will be the same as those of 

25 the texture images. The generated image is output as the sample image. 
The second embodiment will hereafter be explained in detail. 
[0161] 

In a registration means 2100, as registration data to be used for 
comparison of the object, the three-dimensional shape of each object and 
30 image data (texture images) of each object obtained under a plurality of 
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illumination conditions are registered. The registration means 2100 
includes a three-dimensional shape measurement means 2110, a texture 
image photographing means 2120 and a data storage means 21390. 
[0162] 

5 The three-dimensional shape measurement means 2110 

measures the three-dimensional shape of each object by use of a 
three-dimensional shape measurement device such as the device 
disclosed in Japanese Patent Application No.HEIll-123687, and outputs 
the three-dimensional shape data to the data storage means 2130. 
10 [0163] 

The texture image photographing means 2120 actually sets 
illumination conditions which are equivalent to those set and output by 
the illumination condition variation means 231 of the first embodiment, 
and photographs each object under the illumination conditions. For 

15 example, a hemispherical scaffold is set up in front of the registered 
object and an appropriate number of illumination lamps are fixed to the 
scaffold. Concretely, in the angles ( 9 , <t> ) shown in Fig. 4 with respect 
to the object, the lamps are fixed to points corresponding to angles from 
-90° to 90° at intervals of 15° for example, and the object is 

20 photographed repeatedly by successively turning each lamp on. It is 
also possible to photograph the registered object repeatedly moving a 
lamp by a manipulator and turning the lamp on repeatedly. The texture 
images obtained above are output to the data storage means 2130. 
[0164] 

25 The data storage means 2130 stores the three-dimensional 

shape data of the objects supplied from the three-dimensional shape 
measurement means 2110 and the texture images supplied from the 
texture image photographing means 2120. The registered data are read 
out at the time of the comparison process by the comparison means 2200. 

30 [0165] 
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The comparison means 2200 includes a photographing means 
2210, a position/pose estimation means 2220, an illumination correction 
means 2230, an image comparison means 2240 and a judgment means 
2250. 
5 [0166] 

The photographing means 2210 which is provided with a camera, 
a video camera, etc. photographs a comparison target object and outputs 
the obtained image to the position/pose estimation means 2220, the 
illumination correction means 2230 and the image comparison means 
10 2240 as an input image. 
[0167] 

The position/pose estimation means 2220 estimates 
position/pose parameters, photographing device parameters, etc. at the 
time when the target object was photographed. The position/pose 

15 parameters include translation distance (Tx, Ty, Tz) of the object, 
rotation angles (Rx, Ry, Rz) of the object, the focal length "f of the 
camera, and the viewing angle " a " of the camera, for example. The 
position/pose estimation means 2220 is provided with an interactive 
interface, and the user of the position/pose estimation means 2220 

20 adjusts the position/pose parameters manually watching a display 
screen. 
[0168] 

The illumination correction means 2200 does not include the 
illumination condition variation means 231, and therefore, in the image 
25 generation means 2232, illumination variation images are generated by 
use of, as intensity value of a surface of the object, the texture images 
being photographed by the texture image photographing means 2120. 
[0169] 

Fig. 7 is a block diagram showing the composition and process 
30 flow of an illumination correction means 2230 according to the second 
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embodiment. The illumination correction means 2230 includes an 
image generation means 2232, an illumination variation space 
generation means 2233 and an illumination condition estimation means 
2234. 
5 [0170] 

The image generation means 2232 reads out the 
three-dimensional shape data and the texture images of a comparison 
target object "j" from the data storage means 2130, and generates 
illumination variation images by computer graphics, by use of the 

10 position/pose parameters obtained by the position/pose estimation means 
2220 and each of the texture images. The process can be conducted by 
means of texture mapping, which is a basic function of a computer 
having graphic functions. In the second embodiment, various types of 
camera models such as the pinhole camera model can be employed. The 

15 texture images employed in the second embodiment are actually 
photographed images, therefore, there is no need to generate shadows 
and specularity by computer graphics as in the first embodiment. 
[0171] 

The illumination variation space generation means 2233 
20 calculates illumination variation space by use of the illumination 
variation images generated by the image generation means 2232 
according to the [Equation 3], and the calculated basis vectors are output 
to the illumination condition estimation means 2234 as the illumination 
variation space (basis vectors) ^j. 
25 [0172] 

The illumination condition estimation means 2234 generates, as 
a reference image, an image which is the nearest in the illumination 
variation space to the input image by use of the input image 
according to the [Equation 6], and the generated reference image is 
30 output to the image comparison means 2240. 
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[0173] 

The image comparison means 2240 calculates an evaluation 
value concerning the similarity between the input image and the 
generated reference image. 
5 [0174] 

The judgment means 2250 judges whether or not the target 
image is the registered image by use of the calculated evaluation value 
and a threshold value. When a plurality of objects have been registered, 
the reference image generation by the illumination correction means 

10 2230 and the evaluation value calculation by the image comparison 
means 2240 are conducted repeatedly. In this case, the judgment means 
2250 can judge which registered object is the most similar to the target 
object. It is also possible to search for one or more registered objects 
that are similar to the target object by finding registered objects having 

15 more than certain evaluation value. 
[0175] 

According to the second embodiment of the present invention, 
more time and effort become necessary in the data registration step in 
comparison with the first embodiment since the texture images have to 
20 be obtained by actually photographing objects to be registered to the data 
storage means 2130. However, numerical calculations for the 
approximation in the low-dimensional space and the processes for 
generating shadows and specularity can be omitted since the actually 
photographed texture images are used for the illumination correction 
25 process, and therefore, processing time necessary for the comparison step 
can be shortened in comparison with the first embodiment. 
[0176] 
(Third Embodiment) 

In the following, a detailed description will be given of a third 
30 embodiment of the present invention. Fig. 8 is a block diagram showing 
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the composition and process flow of an image comparison device in 
accordance with the third embodiment of the present invention. In this 
embodiment, instead of the three-dimensional shape measurement 
means 3110 which measures the three-dimensional shapes of all objects 
5 when registering a plurality of objects in the first embodiment, by 
measuring the three-dimensional shapes of one or a small number of 
objects, an average shape generation means 3150 generates an average 
three-dimensional shape of the measured objects. The third 
embodiment is different from the first embodiment in that shapes of all 
10 the comparison target objects are not measured, and that a comparison 
means 3200 uses the three-dimensional shape of the average shape. 
[0177] 

The image comparison device of the third embodiment can 
perform the position/pose estimation process and the illumination 
15 correction process by use of the representative three-dimensional shape 
data instead of measuring and using the three-dimensional shapes of all 
objects in the case where the shapes of the objects are similar. 
[0178] 

A registration means 3100 includes a three-dimensional shape 
20 measurement means 3110, an average shape generation means 3150, a 
reflectance measurement means 3120 and a data storage means 3130. 
[0179] 

The three-dimensional shape measurement means 3110 
measures three-dimensional shapes of objects 1 and 2, by use of such a 
25 three-dimensional shape measurement device as disclosed in Japanese 
Patent Application No.HEI 11- 123687. 
[0180] 

The average shape generation means 3150 translates the 
three-dimensional shape data of the registered object 1 or 2 so that the 
30 barycenters of the objects 1 and 2 will overlap one another as shown in 
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Fig. 9(a), sets sections that are perpendicular to the z-axis at appropriate 
intervals, and calculates average shapes on each of the sections. As 
shown in Fig. 9(b), a line being average calculation line is drawn outward 
from the barycenter of the object on the section, and intersection points 
5 Pi and P2 of the average calculation line with shapes of the objects 1 and 
2 are obtained. The three-dimensional coordinates of a point P m of the 
average shape are obtained by averaging three-dimensional coordinates 
(xi, yi, zi) and (x2, y2, Z2) of the surface points Pi and P2 as expressed by 
[Equation 8]: 
10 [0181] 

[Equation 8] 

' *1+*2 ^l+O ...(g) 

I 2 ' 2 ' 2 J 
[0182] 

By performing the above process by repeating the above 
15 calculation around the barycenter at proper intervals, the average shape 
of the objects 1 and 2 can be generated. Thereby, the generated average 
shape can be output to the data storage means 3130. 
[0183] 

The reflectance measurement means 3120 measures the 
20 reflectance of the surface of each object 1 and 2 corresponding to the 
three-dimensional shape. If the three-dimensional shape measurement 
device of Japanese Patent Application No.HEI 11- 123687 is employed, 
color information on the surface of the object can be measured 
simultaneously with the three-dimensional shape. In the following, the 
25 color information is substituted for the reflectance data. The 
correspondence between each point of the average three-dimensional 
shape and the color information of each registered object can be set as 
follows. That is, if we express the three-dimensional shape data of the 
objects 1 and 2 which have been used for the calculation of the 
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three-dimensional coordinates of the average three-dimensional shape 
P m as Pi and P2, reflectance data of the object 1 corresponding to the P m 
is the reflectance corresponding to the Pi, and in the same way, 
reflectance data of the object 2 corresponding to the average shape P m is 
5 the reflectance corresponding to P2. 
[0184] 

The data storage means 3130 stores the average shape data 
input from the three-dimensional shape measurement means 3110 and 
the reflectance of the objects 1 and 2 input from the reflectance 
10 measurement means 3120. 
[0185] 

The processing of the comparison means 3200 is basically the 
same as that of the comparison means 200 of the first embodiment, 
except that the average three-dimensional shape data read out from the 
15 data storage means 3130 is changed to the average shape of each of 
objects 1 and 2. 
[0186] 

While an embodiment in which an average shape of two objects 
is stored when two objects are registered has been explained above, this 
20 embodiment is only for an example. An embodiment in which the 
number of objects is more than three and arbitrary number thereof are 
calculated for obtaining the average shape is also possible by the same 
processing. 
[0187] 

25 (Fourth Embodiment) 

In the following, a detailed description will be given of a fourth 
embodiment of the present invention. In the fourth embodiment, 
human face is used as an example of a comparison target object. Fig. 10 
is a block diagram showing the composition and process flow of an image 

30 comparison device in accordance with this embodiment. Differently 
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from the image comparison device of the first embodiment, a registration 
means 4100 is further provided with a feature point extraction means 
4140 which extracts feature points where intensity etc. changes steeply 
from each object and outputs the extracted points as the feature point 
5 positions. A data storage means 4130 of the fourth embodiment also 
stores the feature point positions. In a comparison means 4200 of the 
fourth embodiment, a position/pose estimation means 4220 reads out the 
feature point positions from the data storage means 4130 and 
automatically executes the position/pose estimation. 
10 [0188] 

According to this embodiment, if feature point positions of the 
comparison target object in the input image can be extracted and 
three-dimensional coordinates of the feature points can be obtained 
based on the registered data, parameters concerning the position/pose of 

15 the object in the input image and parameters concerning the 
photographing device can be obtained automatically by use of the 
positions of the feature points on the input image and the 
three-dimensional coordinates. As a method for automatically obtaining 
the parameters concerning the position/pose and the photographing 

20 device when positions of points in an image and three-dimensional 
coordinates of the points are known, a camera calibration method can be 
used. 
[0189] 

In short, it is a method that, instead of comparing images, the 
25 position of the points or areas having features (hereinafter referred to as 
feature points) where intensity changes steeply etc. on an image is 
detected in the input image and CG images, and thereby, parameters 
concerning the position/pose and the photographing device are calculated 
by selecting a CG image whose feature point positions are the nearest to 
30 those of the input image. 
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[0190] 

It is also possible to determine the position/pose of the target 
object by use of the relationship between the feature point positions in 
the input image. In a document^ Shinn-Ying Ho and Hui-Ling Huang 
5 "An analytic solution for the pose determination of human faces from a 
monocular image", Pattern Recognition Letters, Vol.19, pp. 1045- 1054 
(1998), feature points such as the tails of eyes and the ends of lips are 
used when human faces are used as a comparison target object, and the 
position/pose of an object is determined by use of the relationship 
10 between the feature points such that a line connecting two eye-feature 
points is parallel to a line connecting two mouth-feature points, etc. 
Hereinafter, a detailed description will be given of this embodiment. 
[0191] 

The registration means 4100 includes a three-dimensional shape 
15 measurement means 4110, a reflectance measurement means 4120, a 
feature point extraction means 4140 and a data storage means 4130. In 
the registration means 4100, the three-dimensional shape and 
reflectance of each object are measured as the registered data to be used 
for the object comparison, and three-dimensional coordinates of feature 
20 points of the object are obtained by use of the measured 
three-dimensional shape and reflectance. The three-dimensional shape, 
the reflectance or color information and the feature point positions are 
registered. 
[0192] 

25 The three-dimensional shape measurement means 4110, which 

is provided with a three-dimensional shape measurement device, 
measures the three-dimensional shape of each object. While the 
three-dimensional shape measurement device of Japanese Patent 
Application No.HEI 11- 123687 is employed in the fourth embodiment as 

30 an example, other devices can of course be used. 
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[0193] 

The reflectance measurement means 4120 measures the 
reflectance of the surface of the object corresponding to the 
three-dimensional shape. If the three-dimensional shape measurement 
5 device of Japanese Patent Application No.HEIll-123687 is employed, 
color information on the surface of the object can be measured 
simultaneously with the three-dimensional shape. The color 
information obtained by the device will hereafter be substituted for the 
reflectance. 
10 [0194] 

The feature point extraction means 4140 detects points or areas 
where intensity etc. changes steeply (hereinafter referred to as feature 
points) of the object on an image, and outputs the three-dimensional 
coordinates of the feature points (feature point positions) in the data 

15 storage means 4130. For example, when human faces are comparison 
target objects, parts where reflectance changes steeply such as tails of 
eyes, lips, etc., parts where the three-dimensional shape changes steeply 
such as tip of nose etc., are detected. The feature point detection 
process can be conducted by manual operation. For the automatic 

20 feature point extraction process, various methods including those 
disclosed in the aforementioned Japanese Patent No.2872776: "face 
image comparison device" and the aforementioned Japanese Patent 
Application Laid-Open No.HEI6-168317: "personal identification device" 
can be employed. In this embodiment, twelve points (0 ~ 11) shown in 

25 Fig. 12 are used as the feature points. The definition of the feature 
points can of course be altered depending on what the comparison target 
object is. In the following explanation, the feature point positions being 
three-dimensional coordinates will be expressed by use of vectors Ai = (xi, 
yi, Zi) (i = 0, 1, • • • , 11). 

30 [0195] 
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The data storage means 4130 stores the three-dimensional 
shapes, the reflectance and the feature point positions of the registered 
objects. The registered data is read out when the comparison step is 
conducted by the comparison means 4200. 
5 [0196] 

The comparison means 4200 includes a photographing means 
4210, a position/pose estimation means 4220, an illumination correction 
means 4230, an image comparison means 4240 and a judgment means 
4250. 
10 [0197] 

The photographing means 4210, which is provided with a 
camera, a video camera, etc., photographs the comparison target object 
and thereby obtains the input image. The input image obtained by the 
photographing means 4210 is supplied to the position/pose estimation 
15 means 4220, the illumination correction means 4230 and the image 
comparison means 4240. 
[0198] 

The position/pose estimation means 4220 estimates the 
parameters concerning the position/pose of the object and parameters 
20 concerning the photographing device being the condition at the time 
when the target object was photographed. Fig. 11 is a block diagram 
showing the composition and process flow of the position/pose estimation 
means 4220. The position/pose estimation means 4220 includes an 
input image feature point extraction means 4221 and a position/pose 
25 calculation means 4222. 
[0199] 

The input image feature point extraction means 4221 extracts, 
from the input image, feature points being the same position as feature 
point vectors Ai extracted by the feature point extraction means 4140 of 
30 the registration means 4100, and outputs position vectors on the image 
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Bi = (iii, Vi) (i = 0, 1, • * • , 11) to the position/pose calculation means 4222 
as input image feature point positions. The input image feature point 
extraction process can be conducted by manual operation of an operator 
seeing the input image on the screen. The methods employed by the 
5 feature point extraction means 4140 such as the methods disclosed in the 
aforementioned Japanese Patent Application Laid-Open 
N0.HEI6- 168317: "personal identification device" can be used. 
[0200] 

While human face comparison is taken as an example in this 
10 embodiment, the image comparison device of the fourth embodiment can 
be used for the comparison of polyhedron-like objects, for example, by 
using apexes of the objects as the feature points. Edges are extracted 
from each object in the image, and the apexes of the object are detected 
as intersection points of the edges. When some characteristic patterns 
15 exist on the surfaces of the objects, the positions of the patterns can be 
used. 
[0201] 

The position/pose calculation means 4222 calculates parameters 
concerning the position/pose of the target object in the input image and 

20 parameters concerning the photographing device by use of the feature 
point positions read out from the data storage means 4130 and the input 
image feature point positions input by the input image feature point 
extraction means 4221, and outputs the calculated parameters to the 
illumination correction means 4230 as the position/pose. For this 

25 calculation various methods, including a method disclosed in a 
document: Roger Y. Tsai "An Efficient and Accurate Camera Calibration 
Technique for 3D Machine Vision", Proc. CVPR '86, pp364-374 (1986), 
can be employed. 
[0202] 

30 In this embodiment, the position/pose parameters employed in 
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10 



15 



20 



the fourth embodiment include, for example, the translation distance (Tx, 
Ty, Tz) of the object, rotation angles (Rx, Ry, Rz) of the object around the 
x-axis, y-axis and z-axis and the focal length "f of the camera. The 
pinhole camera model is employed as the camera model. The 
relationship between the feature point position vector Ai and the input 
image feature point position vector Bi can be expressed by the following 
[Equation 9], 

[0203] 
[Equation 9] 

-(9) 
[0204] 

where "a", "b" and "c" are defined as following [Equation 10]: 

[0205] 
[Equation 10] 
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[0206] 

where "R" is the following rotation matrix expressed by the [Equation 
11]: 

[0207] 
[Equation 11] 

sR x sin R : + sin R x sin R y cosR z 
cosR x 
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cosR x cos R y 
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[0208] 

25 The position/pose parameters Rx, Ry, Rz, Tx, Ty, Tz, f by 
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optimization calculation so that the sum of the differences between the 
value obtained by the [Equation 9] and the input image feature point 
positions with respect to the 12 feature points will be the smallest. The 
optimization calculation can be executed in various ways. The obtained 
5 Rx, Ry, Rz, Tx, Ty, Tz, fare supplied to the illumination correction means 
4230 as position/pose parameters. The definitions of the position/pose 
parameters and the camera model and the calculation method explained 
above are only examples, and thus, the position/pose calculation process 
can be conducted according to various methods. 
10 [0209] 

The illumination correction means 4230 generates an image, as 
a reference image, of an object in the same position/pose as an object in 
the input image under an illumination condition that is the most similar 
to that of the input image, using the parameters obtained by the 
15 position/pose estimation means 4220. 
[0210] 

The image comparison means 4240 calculates an evaluation 
value concerning the similarity between the input image and the 
generated reference image. Various calculation methods can be 
20 employed for the evaluation. For example, the sum of squares of pixel 
intensity differences which has been shown in the [Equation 7] can be 
used as the evaluation value. 
[0211] 

The judgment means 4250 executes judgment whether or not 
25 the comparison target image is the registered image by use of the 
calculated evaluation value and a threshold value. When a plurality of 
objects have been registered as the registered objects, the reference 
image generation by the illumination correction means 4230 and the 
evaluation value calculation by the image comparison means 4240 are 
30 conducted repeatedly for each registered object. It is also possible to 
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search for one or more registered objects that are similar to the target 
object by finding registered objects giving certain evaluation values. 
[0212] 
(Fifth Embodiment) 
5 In the following, a detailed description will be given of an image 

comparison device in accordance with a fifth embodiment of the present 
invention. In this embodiment, the objects to be registered are industry 
products etc. whose design drawings of the shapes have been saved as 
CAD data and whose surface painting specifications etc. have been 

10 designated and written in drawings. Differently from the first 
embodiment, a three-dimensional shape measurement means 110 of the 
fifth embodiment obtains the three-dimensional shape data from the 
CAD data of design drawings, and a reflectance measurement means 120 
of the fifth embodiment obtains the reflectance data by reading drawings. 

15 The image comparison device of the fifth embodiment can preferably be 
applied to cases where the three-dimensional shape measurement by use 
of an ordinary three-dimensional shape measurement device is difficult 
(for example, when the registered objects are houses, buildings, etc.) and 
the three-dimensional shape is obtained by actually making a survey or 

20 measurement. 
[0213] 

The three-dimensional shape measurement means 110 reads the 
CAD data of the of the design drawings, converts the CAD data into a 
data format that can be handled by the comparison means 200, and 
25 outputs the converted data to the data storage means 130. 
[0214] 

The reflectance measurement means 120 reads data concerning 
the colors, surface finishing conditions, etc. of parts of the objects from 
the design drawings, converts the data into reflectance data, and outputs 
30 the data to the data storage means 130. 
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[0215] 

The data storage means 130 stores the three-dimensional shape 
data obtained by the three-dimensional shape measurement means 110 
and the reflectance data obtained by the reflectance measurement means 
5 120. 

[0216] 

The comparison means 200 of the fifth embodiment conducts the 
comparison step in the same way as the first embodiment. 
[0217] 

10 As described in the above embodiment, the present invention 

can be applied to general objects. Especially, the present invention can 
be applied to the comparison of the type and form of automobiles, and 
also the comparison of human faces. 
[0218] 

15 As set forth hereinabove, while a detailed description has been 

given of the embodiments of the present invention with reference to the 
drawings, it is not to be restricted by those embodiments, and change or 
modification of the embodiment is to be appreciated without departing 
from the subject matter of the present invention. The present invention 

20 can also be implemented by use of a computer program. 
[0219] 
[Effect of the Invention] 

According to the present invention, as it is clear from the above 
explanation, three-dimensional shapes of objects and reflectance of 

25 surface of the objects or images under appropriate conditions of 
illumination are measured only by the registration means, and 
photographing device such as video camera for photographing general 
two-dimensional images is enough as a photographing means in a 
comparison means, and thereby, a practical device can be constructed 

30 without a need of a three-dimensional shape measurement device in the 
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comparison step. 
[0220] 

Further, three-dimensional shapes are registered, and therefore, 
perfect correction is possible with respect to the change of the 
5 three-dimensional position/pose in the input images. The present 
invention can be applied to the target objects whose surface reflectance 
properties are not perfect scattering, and also applied to the target 
objects which cast shadows due to light occlusion by other parts, 
specularity, etc. occurring on the surfaces, and therefore, enough 
10 correction can be executed with respect to the variation of illumination 
conditions. Thus, the present invention can be applied to comparison of 
general objects in wider range than the above Illumination Subspace 
Method, or the Sampling Method. 
[0221] 

15 Moreover, in the comparison step, illumination variation images 

can automatically be generated by use of registered three-dimensional 
shapes and reflectance, and accordingly, the registration step can be 
performed easily since photographing a lot of images is not needed. 
Furthermore, by use of the cumulative contribution, an enough 

20 dimension number of subspace as an approximation of the illumination 
variation space can be obtained, and therefore, the amount of 
calculations necessary for the illumination correction process can be 
reduced without deteriorating the accuracy of the image comparison. 
[Brief Description of the Drawings] 

25 [Fig. 1] 

A block diagram showing the composition and process flow of an 
image comparison process in accordance with a first embodiment of the 
present invention. 
[Fig. 2] 

30 A constitutive diagram in accordance with the first embodiment 
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of the present invention. 
[Fig. 3] 

A block diagram showing the composition and process flow of an 
illumination correction means 230 in accordance with the first 
5 embodiment. 
[Fig. 4] 

A diagram for explaining angles indicating the direction of 
illuminations which are used for the setting of illumination conditions 
with respect to an object. 
10 [Fig. 5] 

A diagram showing an example of an object comparison device 
using images. 
[Fig. 6] 

A block diagram showing the composition and process flow of an 
15 image comparison device in accordance with a second embodiment of the 
present invention. 
[Fig. 7] 

A block diagram showing the composition and process flow of an 
illumination correction means 2230 in accordance with the second 
20 embodiment of the present invention. 
[Fig. 8] 

A block diagram showing the composition and process flow of an 
image comparison device in accordance with a third embodiment of the 
present invention. 
25 [Fig. 9] 

Diagrams for explaining a generation method of an average 

shape. 

[Fig. 10] 

A block diagram showing the composition and process flow of an 
30 image comparison device in accordance with a fourth embodiment of the 
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present invention. 
[Fig. 11] 

A block diagram showing the composition and process flow of a 
position/pose estimation means 4220 in accordance with the fourth 
5 embodiment of the present invention. 
[Fig. 12] 

A diagram showing an example of points employed as feature 
points of a target object. 
[Fig. 13] 

10 A block diagram for explaining, as an example of a conventional 

image comparison technique, the composition of a technique in which 
only two-dimensional images are employed in both the registration step 
and the comparison step. 
[Fig. 14] 

15 A block diagram for explaining, as an example of a conventional 

image comparison technique, the composition of a technique in which 
three-dimensional shapes are measured in both the registration step and 
the comparison step. 
[Fig. 15] 

20 A block diagram for explaining, as an example of a conventional 

image comparison technique, the composition of a technique in which 
two-dimensional images are employed in both the registration step and 
the comparison step, and standard three-dimension shape is employed in 
the position/pose correction step. 

25 [Fig. 16] 

A block diagram for explaining, as an example of a conventional 
image comparison technique, the composition of a technique in which 
images are photographed under a lot of position/pose conditions in the 
registration step and recognition is performed. 

30 [Fig. 17] 
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A block diagram for explaining, as an example of a conventional 
image comparison technique, the composition of a technique in which 
two-dimensional images are photographed under a plurality of 
illumination conditions in the registration step, and illumination 
5 condition correction is performed. 
[Fig. 18] 

A schematic block diagram for explaining, as an example of a 
conventional image comparison technique, the composition of a technique 
in which two-dimensional images are photographed under a plurality of 
10 illumination conditions in the registration step, and illumination 
condition correction is performed. 

[Description of Codes] 
100 Registration means 

110 Three-dimentional shape measurement means 
15 120 Reflectance measurement means 

130 Data storage means 

200 Comparison means 

210 Photographing means 

220 Position/pose estimation means 
20 230 Illumination correction means 

231 Illumination condition variation means 

232 Image generation means 

233 Illumination variation space generation means 

234 Illumination condition estimation means 
25 240 Image comparison means 

250 Judgment means 
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[Title of Document] Abstract 
[Abstract] 

[Problem] To provide an image comparison device, an image 
comparison method and a memory medium in which a program thereof is 
5 stored, where three-dimension shapes are not needed to be measured in 
a comparison step, and identification of whether or not a target object is 
a registered object, comparison of which registered object is the target 
object, and a search of similar objects are performed by use of images 
photographed with various position/pose of objects and under various 

10 illumination conditions. 

[Means of Solving] A registration means 100 measures the 

three-dimensional shape and surface reflectance of an object to be 
registered and stores them as registered data. In a comparison means 
200, a photographing means 210 photographs an input image. A 

15 position/pose estimation means 220 estimates the position/pose of the 
object in the input image by use of the registered three-dimension shape 
and reflectance. An illumination correction means 230 generates, as a 
reference image, an image of the object in the same position/pose as the 
object in the input image and under the same illumination condition as 

20 in the input image by use of the registered three-dimensional shape and 
reflectance. An image comparison means 240 calculates an evaluation 
value concerning the similarity between the reference image and the 
input image. A judgment means 250 performs comparison judgment 
based on the evaluation value. 

25 [Selected Drawing] Fig. 1 
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